Not just the weird billing, which has been annoying the last month I've been trying it, but they have had a couple incidents lately where I feel like I got ripped off because their performance was so spotty and yet I was getting billed for it. https://status.blacksmith.sh/
I had a similar issue with Blacksmith, though it was something called "Sticky disk" that caused unexpected costs. I'm still not even sure what it is.
It was disappointing to see -- I've used Depot in the past and will stick with them where possible.
While I think it’s outrageous to send invoices to folks who aren’t paying customers… I’m a little surprised the service even has a niche (I’m old). I guess, with ubiquity of containers in our modern workflows, it seems strange to pay a service for what I assume is a dedicated, or well provisioned VPS, just to run CI. Hell you can probably get Jenkins (further showing my age here) running in less than an hour with Claude, GPT, or Deepseek on an obscure provider that offers cheap bare metal instances.
And for anyone who hasn’t used bare metal instead of over provisioned VPS for services the performance gap is noteworthy and substantial. Yeah, there is some risk because you have to worry about outages, upgrades, and configuration but for something like CI where there’s near zero data loss risk… it seems well worth it if performance of your CI/CD infrastructure is really an issue.
Sounds like a business partner who will squeeze you again later
> Well, we tried switching back to GitHub Actions, and… yeah it still sucks. Blacksmith has grown explosively because it makes an increasingly frustrating bottleneck in the dev cycle faster.
Where's the "grown explosively" bit coming from? Google doesn't have growth news from 2026, only early 2025.
$1000 in a month? What on earth are you doing in CI? We're on Warp build (50% cheaper instead of 33%) with 6 people hitting it pretty hard across a few repos and only reach $150/mo. That's with building rust - a worse case than anything you're using for build time.
1. I'm not sure if blacksmith offers larger runners, but you if you're using them you have checked that bigger runners are worth the squeeze? A 2x runner does not mean a 2x faster build - I had a goal and sized CI according to that.
2. Caching. Nx can do this for your TS, provided your code is decomposed into packages.
3. At a previous job I had a post-run timings job. GH markdown supports Mermaid; I used a gannt chart to express it. I don't remember if the GH api supports getting timing for the current wf - so it may have been a second workflow.
The first is just a little legwork, the rest your agent should be able to do in 5 minutes.
> This question is for us: will we keep using Blacksmith, despite them giving us an unpleasant surprise and a prickly support exchange?
Interesting question, and I think it makes sense they’ve chosen to be pragmatic.
However, I wonder if Blacksmith get bitten in the arse by this. Hopefully that changes their behaviour but, like many startups, they might simply fail.
So I suppose the trick becomes to keep using the service without getting locked into it until it becomes clearer whether they will succeed or not, then perhaps you can consider taking advantage of platform features.
Even then, I don’t know how much I trust Blacksmith or would want to make it hard for myself to move away from them.
And on GitHub actions: Microsoft are very good at owning the platform and then making products and features that are just useful enough that it’s not worth switching to a better alternative but absolutely no more. GitHub Actions is an obvious example, Teams is another, but the list is long. To me it reads as a more modern variant of the anticompetitive behaviour of the 1990s. It sets up enough of a barrier to keep others out, and kills innovation. I’m not a fan.
> In order to use the Blacksmith Software Inc Service, You must set up an account. During the account setup process, You will be required to connect your GitHub account and install Blacksmith’s GitHub integration in your org, and add a valid payment method, such as a credit card, which will be processed through Stripe. Alternatively, for larger contracts, You may request to be billed via invoice.
> By providing payment information, You authorize us to charge Your credit card for usage fees or, in the case of invoice-based contracts, agree to make timely payments as specified in the invoicing terms.
Unless this guy had a larger contract and requested to be billed via invoice, this is a violation of terms and he should tell them to stuff it.
I can see their logic - instead of breaking people's builds, they are being the nice guys and letting you pay them back later. BUT, any time you break convention, you've dipped into your trust budget (even if you communicate it way more clearly than was apparently done here.)
Can someone explain this statements:
> While amusingly as of June 8 Blacksmith’s terms implied that their right to bill you is contingent on you providing payment information, a SaaS app certainly could have terms that obligate users to pay for unexpected overage when on a free trial.
> And let’s be clear: our agents run a lot of CI jobs, so we did expect to hit the limits of the free plan. We used the service and got value for it. So it’s not inherently dishonest, just surprising. My read is that they can do this.
So I read this that the terms say "blacksmith will only bill you if you provide payment information" but then they say that blacksmith can bill you if you don't provide payment information. That seems to contradict the terms, which I would assume underlie the "contract" that you agree to when agreeing to the terms.
I can understand not wanting to kill someone's CI in the middle of active development, but then the product needs a very explicit consent step before converting free minutes into deb
This reminds me of OpenAI which allows you to overcommit on prepaid tokens and then tries to force you into paying overpaid charges. Only they really can’t. You can’t make someone pay for a service they didn’t agree to with billing that doesn’t exist. I wish OpenAI best of luck with their shenanigans
I’m running my own VPS with Woodpecker, paying for someone to do it for you seems absurd to me.
Good timing, thanks, just scratched Blacksmith off my shortlist. Depot is the current front runner.
>Blacksmith helps us build faster, and once we agreed to pay for the (actually useful) service, their support got friendlier. So… we’ll probably switch back.
well, that was one of the most disappointing conclusions that i have read recently.
i am almost more frustrated with this decision than i am with the billing shenanigans. please do not reward this scummy behavior.
Dark pattern, shitty behaviour.
Nothing to excuse
[dead]
Avrea.com founder here, if you'd like to try an alternative that won't surprise you with the bill, I'd be happy to give you some more credits for a trial.
Why am I not surprised that its a YC startup? Lately, being a YC startup seems to have become a negative signal for me, far too many grifters are getting funded by YC, it seems.
This seems like the wrong tactic, even in a money grabbing sense.
Either say one month free and end after that, or end after the free data amount. If the customer finds value they will sign up and be less likely to look at their credit card bills.
This just pisses people off and puts the cost front and centre so they can't ignore it.
Holy shit, all that and then they paid? Worse, decided to use this product on an ongoing basis? I'm not sure who I hate more, Blacksmith for doing this or OP for being such a doormat about it.
You should give Depot a try :)
I would understand this coming from a consumer, but surely your company read the contract you signed?
I was thinking of checking out blacksmith for my own startup but yikes.. this kind of surprise gotcha is a major dealbreaker for me.
It's a bizarre way to run a SaaS and their website in migraine-inducing.
Ah, too risky to try for small operators. Good to know. Thanks for the fair warning.
BlackSmith should get in this thread and explain themselves.
Also, can the author tell us how much this would have cost on GH actions?
This is another level up (down?) from the standard "gotcha, b####!" post-usage metered cloud service billing that requires a credit card up front which is the main business model for a surprising number of internet companies. They can keep doing it because a significant percentage of their business is from startups that are burning through someone else's money.
But it's interesting that the only real difference is that they let you proceed without putting your credit card info in.
Ignoring at least 2 messages does not in-fact make a surprise if you just don't like what the messages said.
Infisical does this too. They don't make it clear that they charge you for projects and machine identities upfront, and then you get slammed with a $800 bill on your first month.
[dead]
Why do people pay for runner saas at all? Theyre not exactly hard to DIY
Presumably they wanted feature parity with Github Actions. If you have lots of workflows already defined in Github Actions DSL, you're not going to want to port them. And even if you do port them, the activities available in the place you port them aren't going to be the same as Github Actions; there won't be feature parity.
Standard growth-hack douche-baggery.
You'd think the YC mentors would give advice like "Don't be a dick to your potential customers".
Also I'd tell them where to stick their invoice.
Skip the middleman, just use AWS with runs-on.com (I'm the developer). Better instance choice, much cheaper, and you can setup an AWS budget alert.
TBF, I believe Blacksmith runs their own hardware, so they aren’t just middlemen. I still think the billing thing feels scummy, though.
[dead]
If you want a 'fixed' cost option for GitHub self hosted runners on your AWS I can't recommend https://runs-on.com/ enough.
A couple caveats. Doesn't support EC2 Macs, so you can't use this to build iOS apps. (Understandably. EC2 Macs are a pain to support and can't reasonably be scaled elastically due to a combination of Apple licensing restrictions and Mac hardware being totally unsuitable to the datacenter.) Also, this is a EU company which may be a problem for US-based customers.
Thanks for the warning. I would have chosen to be pragmatic as well but with another vendor. While I applaud your generosity with trust mine would have been evaporated at the first invoice.
Whats worse, if people stay with them after such a scummy behavior, they will keep doing it.
> This question is for us: will we keep using Blacksmith, despite them giving us an unpleasant surprise and a prickly support exchange?
Well, there are other drop-in GHA runner services, so I wouldn’t see why anyone would be tied into a specific provider.
Namespace.so are one and my experience with their support has been incredibly positive. Great team there.
Honourable mention to WarpBuild as well, who I used before them.
Hugo from Namespace here. Thanks for the kind words! We try really hard to be there for teams, appreciate you recognizing that.
How exactly would BlackSmith enforce the overdue payment? By sending the user to court?
Unlikely. But it is likely they will need to pay before resuming usage as a paying customer.
They could if they feel it's worthwhile. Most companies don't, but most companies don't do most of the stuff mentioned in this article, because they're lazy. If they're not lazy they can absolutely follow up any unpaid debt in court, no matter whether you tried to use a virtual credit card or anything like that.
In German-speaking (DACH) countries the companies aren't lazy and they will take you to court and the court will make you pay all legal and court fees as well as the debt. It's a near certainty they will bother. In the USA you're hoping they won't bother and they'll be satisfied with just banning you as a customer. I think this is because each party pays their own legal fees in the USA.
Yes, civil legal proceedings (and/or hiring a collections agency) are generally how debts are pursued in the United States.
That's a typical discrepancy between how US and EU sees civil agreements.
In Europe it's perfectly normal to be bound by terms of a paid service. I would never expect to avoid being liable for payments for services rendered only because I didn’t enter a cc number before exceeding free-tier limits.
Even in the comments below people are stating that this bill is valid only if they want to continue using the service.
I dunno. I'm in the US and I feel that those saying they wouldn't expect to be liable for services they consume beyond free tier limits sound awfully entitled.
I mean, it's a sleazy practice, especially considering how they word their emails, but I wouldn't expect something for nothing just because I hadn't set up a billing mechanism yet.
"I ignored a ton of emails from Blacksmith and then I was completely surprised by something that happened"
Lessons learned:
1) When you spam your users with too many emails (engagement! marketing-thinly-wrapped-as-transactional-information!), they stop reading your emails
2) Read your damn emails
B2B is just businesses fighting over the value chain.
Blacksmith can send as many invoices as they want.
If the other business doesn't think it's worth the cost after the free trial, they don't have to pay the invoices.
I'm old enough to remember that capitalism used to always feel like this. Even worse. Predatory subscriptions, targeting minors in some cases (remember phone ringtones?), toll numbers, unclear fees, surprise bills (my favorite - roaming fees in your home country because the phone stayed logged into the foreign network after an abroad trip). Silicon Valley/YC style startups felt like a breath of fresh air. Generous return policies, subscriptions can be canceled monthly and even do so automatically in a lot of cases (eg disabled credit card). I guess the fact that a lot of what they offered has near zero marginal cost helped. AI changed that part, so we're back to what I would consider the natural state.
The least-resistance path out of such a bad equilibrium is regulation. And they did add a lot of protections in the last decades, which probably helped too. Some would say places like the EU even added too many. But I'm pretty sure that "if you're using our product, you need to pay" would fly even in the most customer-friendly jurisdictions today.
It would, and the EU would actually be worse on this. In many if not all EU countries (DACH at the very least) you'd have to pay because you agreed to pay, and you'd have to pay all court and legal fees you caused as well - while in the US the company is more likely to just forget about it and try for another customer.
I am sure Blacksmith is the easy path if you don't want to do any extra work and I am also sure their solution will scale much better as this is what they do full-time.
However, I can share what we did to ease on our GitHub Actions bills if it helps.
Effectively we have our own runners hooked so that a job is scheduled, a runner picks it up and goes with it. We still use GitHub Actions but our monthly bill is now flat because we pay for a server. It is about 6x cheaper if not more.
The solution is not open source but it boils down to a Go service that orchestrates firecracker vms. All the vms are pre-warmed so there is always a fresh supply of workers to pick jobs of various sizes.
It is basic and it works. We have not had any issues since deployed.
The runners can be anything from 1 cpu 2 GB to 64GB 8 cpus. We can add more worker types in a config file.
I am not exaggerating when I say that we used to pay 1000s per month for this. Now our bills are in the range of a few hundred. Other dev boxes are done in the same way.
As someone who has worked with gitlab for the last 5 years, i find it amazing how over complicated the github flow it for CICD! Spin up a server and add the gitlab agent and done! You need more? Fine spin up a EKS k8 cluster and done!
You can do the same with GitHub Actions. Spin up a server and start the actions runner on it and done! But if you want to dynamically scale your runners in response to load using a Kubernetes cluster, then they provide Action Runner Controller and it's a bit more complicated.
> We still use GitHub Actions but our monthly bill is now flat because we pay for a server.
Caveat here being that GitHub is exploring charging a usage-based fee for self-hosted GitHub Action runners [1]. While they've halted it for now, it's something worth being aware of as you assess your costs. This is probably a drop in the bucket compared to the order of magnitude savings you've described.
[1]: https://github.blog/changelog/2025-12-16-coming-soon-simpler...
Allegedly this is because whenever you have a self-hosted runner, it's actually a Microsoft-hosted runner that is notifying your runner and then polling for it to complete. I wish I was joking.
Thanks. I did not know.
So ive done this before when I deal with predatory and 'free trial' companies.
Go get a $20 gas station credit card. They get THAT card, and whatever name you want to provide.
When they demand $x000 for their free trial, they get.... $20!
And before anyone bemoans their 50+ page onesided "contract" that weasel-words revokes 'free trial'... Sure, they can publish a claim and sell off a debt to "John Q Public". We can see how far they'll get with that.
If they can prove you agreed to the charges, they can follow it up with debt collection companies or lawsuits, and that will cost you a lot more than just paying what you agreed to pay.
Reread the last paragraph.
And let me know how a lawsuit against "John Q Public" will go, along with an email from fakemail.net and a gas station preloaded credit card.
Companies deserve this sort of treatment if they say shit like "FREE TRIAL" and silently convert you to 'you owe money' with no hard limits. And naturally, paragraph 37, sentence 12 includes this disclaimer. That should be blatantly clear, but its not cause they are scammers.
So, fuck'em.
WoW. That's certainly a surprise to me. I'd never expect an invoice after not putting in a card.
I also believe this is totally just a case of "billing and metering is hard, and may actually be a larger engineering effort than your actual service".
I was just looking at them earlier today since our Github actions are slow AF, and while they sounds great, this tells me it'll cost me more time to make sure I babysit it than most other trials.
With most of these, they end, the service stops working, and you have a choice to make: (a) it was worth it sign up, (b) not worth it revert.
Founder of Depot [0] here. Feel free to try us out. We have a real free trial that is time based that doesn’t do odd things like this. Also have usage limits that you can put in place to further clamp down on runaway surprises.
This is where the "billing is hard" explanation and the user trust issue overlap
OpenAI used to do billing like this too.
> I'd never expect an invoice after not putting in a card.
Startups have to innovate to stay alive and YC never discourages this kind of "innovation" if the startup has the potential to turn into big dollars.
Yep, the "Its not fraud/illegal if you make defense against cost more than just paying the cost".
It turns legitimate enterprises and businesses into quazi-illegal scam engines. Like, health clubs or Adobe scam-subscriptions.
A certain number of businesses will just pay an invoice if it looks real and has a business justification. There are scams around this, but also vendors who may be taking advantage of this. If 30% of customers will just start paying the invoices, it's worth it to them to deal with the questions from the rest of them. At least until reputational damage starts.
A very simple example of a scam that's probably happened to you if you own a domain name: as the expiration date approaches you'll get "invoices" from companies for domain name renewal charges. If you read the fine print it will say "this is a solicition for business" but otherwise it looks just like an invoice. Some people will just pay them.
Many years ago I met a woman that ran a "business" that simply invoiced legitimate companies for printer supplies and toner every month. A surprising number of her invoices were paid without question.
That went on for a couple years. Then one day she left town basically in the middle of the night and we never heard from her again.
By reading this comment, you agree, etc., etc.
...boilerplate...
...more boilerplate...
Terms... and conditions...
...limitations...
...liabilities...
You can't start a contract merely by reading a comment, but you can start one by using a commercial service with a pricing model, after being informed that you should read the contract.
You now owe me $500.
The $500 is now overdue.
By reading this comment, you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.
That's incredibly scummy. Article author estimates that "only" 5% of people would expect this outcome for a "try for free", "no credit card required" service, but I think that number is well below 1%.
Can't believe they continued using the service after this. I would refuse to pay (they have no legal basis to require payment, and their own terms of service seems to disagree with their behavior) and find a more ethical provider.
Only the no credit card part is surprising. Something like AWS will bill you for everything not within the free tier and expect YOU to keep track of what's in the free tier. But they ask for your CC upfront.
It's not illegal or even unethical to bill someone who hasn't given their CC, but it's definitely unexpected.
That sounds sketchy AF.
Actually quite common, as the cost to fight them isn't worth the legal fees.
One must be extremely careful when signing off on something as a company representative.
One internal IP lawyer wanted a legal journal subscription, and left the tap running after they left the firm... that one cost $8k if I recall, as the journal sold the delinquent account to a collection agency. Took 3 weeks to verify it wasn't a scam, as the companies usually go quiet without the account number etc.
Some people are wired that way, and run their company on legal cons. Indeed, one doesn't want to have these people around your firm. =3
Ooff, this seems a bit sketchy. We recently looked at GA runner alternatives and decided on https://avrea.com. See if you have a better experience with them :)
Great that you liked using us. Happy to have you onboard.(Avrea founder here)
This reminded me of our experience with Gusto - we signed up to their R&D credit payroll offset service in May last year - their offset fee is calculated as a percentage of the benefit you get. We filed our federal tax return in September, so there naturally wasn't any payroll offsetting until October. They still charged for services they didn't deliver so it was more cash out on day one which really goes against the purpose of the tool. They argued it was in the small print that said "we charge when you sign up" when I ticked a box. I thought that was quite outrageous. I still do.
Can I ask how much did they charge you? for how long
About $330, per month, for 5 months.
I'm paying $60/mo. for a Kubernetes cluster running Forgejo + Forgejo Actions with plenty of parallel CI runners.
oh wow
Yea same with our internal gitlab system with a EKS cluster for runners, maybe $100/month for the full infra setup.
I'd love to see a write up on how you've got this deployed if you have one or are willing to share any repos?
I have a ton of write-up that I haven't published.
It's Talos Linux on Hetzner VMs where all resources are managed via kubenix, and all CI runners are running NixOS.
So... extremely affordable and extremely performant, but very complicated.
Drop me an email if you want the writeup when I get around to making it.
Until then, here's a Forgejo Actions that compiles its own CI runner image:
https://git.shine.town/infra/runners/src/branch/main/.forgej...
No Docker involved in the build process, it's all Nix.
Unfortunately the CI runner itself is still Docker-in-Docker because:
https://codeberg.org/forgejo/discussions/issues/66#issuecomm...
Blacksmith are wrong, but also they’re a YC company- they may be young founders that haven’t run a SaaS before and genuinely don’t know how to handle free trials.
Or: they know exactly how to handle free trials
Doesn't take a genius to figure this one out. If you can't understand this before it becomes a problem, you have no place running a SaaS business.
Give the growth hackers the benefit of the doubt, you reckon?
Yes. They don't gain much from pissing people off who don't have a card to bill.
This doesn't seem like the right way to do business long terms. The off chance that someone actually take you up on it and pay your 'bill', you've destroyed a lot of goodwill and alerted the rest of the tech world of your scammy moves.
Also for those that require a credit card for a free trial, I always use a virtual card and cancel it. It's super fun to watch them cry when they can't actually charge you.
[dead]
They will usually refund you if you end up getting charged because you forgot to cancel. It isn't worth the headache of a chargeback.
Plus they have to pay a fee for chargebacks regardless of whether they think it's valid or not, so strong disincentive.
It's way easier to just not give them a way they can charge you. That way you don't have to deal with a support representative fakely asking you how your weekend was, and who doesn't actually care about your weekend.
Funny, I got a fraud call recently because CrunchyRoll decided to try to renew a subscription I abandoned years ago and the card they have is expired.
I know it wasn't me because I gave up entirely on the service after they changed something about their login systems to reject my password and I could no longer get in. Support wanted me to jump through a lot of hoops and I just refused, choosing instead to just stop doing business there because I wasn't really watching anything at that point anyway.
This was around 2022, mind you, so they tried to renew me after several years with no explanation.
This happens with me every year with Barnes and Noble bookstore. I canceled their annual membership back in like 2018, Starting in 2021 this zombie account started charging me for a renewal. I reached out to support and they disavowed any knowledge of how this could happen, had no record of charging me, and no clue how it could have happened.
Since Amex is nice about allowing someone who charged you in the past - keep charging you even as your card has changed, they allow the charge.
Every year I do a chargeback, every year. Amex can't figure out how to not allow it and B&N has no idea why they charge it. Hmmm...
tl;dr:
"Don't do this. It works, but I don't like it."
It seems like a perfectly cromulent business practice to me, unless they start suing people who didn't give them credit cards.
You use the service. You're told, after awhile, that you've racked up a bill. You keep using the service. You're told your racked up bill is bigger.
And yet, the reason you're using the service after the first bill is because you find it valuable.
You have two choices. Pay up to keep using it, or stop.
The fact that you decided to pay up to keep using it is actually, imo, a pretty good advertisement for the service.
Billing statements disguised as marketing nudges is a cromulent business practice until the SaaS start sending bills to collections.
I think the author is being kind, both to themselves and startup practicing dark patterns. He walks through his own thinking, raises important questions and also gives the benefit of the doubt that I wouldn’t give.
IMHO, the article gets ahead of criticism well: accepting the valid critiques while also confining the weird/lazy ones to downvotes.
> Billing statements disguised as marketing nudges is a cromulent business practice until the SaaS start sending bills to collections.
Well, I should have probably said that instead of sued. But the intent is the same.
> I think the author is being kind, both to themselves and startup practicing dark patterns.
It's unclear it's a dark pattern. The author explicitly states that this particular EULA doesn't allow them to bill without a credit card.
But it's also unclear that this practice will survive the exposure on hacker news.
No, not because it's a "dark pattern" but because it enables customer dark patterns. Run up a bill and then go use something else.
It would be perfectly crumulent if it was explicitly communicated in advance.
There are two possibilities here:
1) They intended a bait-and-switch, where they were going to go after everybody for non-payment. According to the article, they might not even have a leg to stand on here.
2) If you take the quote from the company at face value, they realized that their free quota would be insufficient for conversion for some customers, and decided not to shut off services in the middle of evaluation. In this instance, the bill is a communication in advance -- if you provide a credit card in order to keep using our services, we want to get paid for everything you used after the free limit.
Now, you can argue (and many are) about whether this is a good business practice or not, but it really doesn't matter. After making the front page of hacker news, it's probably not one they're going to continue, simply because now that everybody knows about it, you'll probably have a lot of bad actors doing multiple signups, just to siphon off as much token usage as possible.
What's incromulent about suing people who racked up debts to you?
If you read the other comments, you're certainly an outlier in this belief.
My own answer is that there is nothing (incromulent? uncromulent? well, anyway, not cromulent) about suing people who knowingly and deliberately racked up debts with you, but that common business practices, including the overwhelming abundance of free services everywhere in every product category, and the ability to immediately shut off internet services when you aren't paid, lead to sort of a gestalt of expectations about how things are done.
The article itself says that the terms of service only allow billing if a payment method have been provided, so suing absent that provision would probably be a non-starter anyway.
On the bright? side, suing would definitely keep them on the front page of hacker news longer.
I find that exploiting a difference between expectations and reality is a common way to make money, but I'm no good at it myself. I'm cursed with engineer-brain instead of business-brain.
> exploiting a difference between expectations and reality is a common way to make money
Maybe? Or maybe expectations catch up with reality? I mean, does anybody really go into a car dealership thinking the dealer isn't going to do their best to rip them off? Or believe that "their" real estate agent is "on their side?"
I think most people just accept that the world is full of rip-off artists.
Which is kind of sad, really. If one person pushes back on bad billing, the company still makes money, even if it has to do a refund.
But if everybody were to always push back? It wouldn't be worth it to try to tack on extra fees for non-rendered services.
Yes, lots of people who don't frequent HN think their real estate agent is on their side, and think car dealerships just sell cars for a fair price. They might even think politicians make laws to make society better.
That's why all these groups of people are able to make so much money. If everyone expected a car dealer to rip them off, car dealers wouldn't be able to rip anyone off and there wouldn't be so many of them.
Unpleaseant experience and all, but sounds like complaining about not reading the full terms and then being shocked after the first bill, which is ok if you are buying a personal service, but that's a no-no in business.
The argument of "like many early startup do, we oversaw this and ignored that" doesn't really make it better.
Why would it be ok if you're buying a personal service?
Most countries have consumer protection laws, but they're only for consumers, due to the usual power imbalance. Power is expected to be balanced in B2B transactions - your lawyers spent as much time ensuring you weren't scammed as theirs spent trying to scam you - so there's no protection law about that. You get whatever the contract says you get.
Having been a happy customer of Blacksmith, Depot, and Ubicloud at various points (all three of which offer GitHub Actions runners at cheaper prices than GitHub themselves), I agree the billing here is weird. It's worth pointing out, though, that you have to use a lot of CI minutes to incur a $1000 bill. You're well outside of hobbyist levels, and that same level of usage would have previously cost double from GitHub themselves. This a business with serious compute needs, being asked to pay real money.
Wonder if they'd be better served $$$ wise to use dedicated servers or similar?
Yes, having a performant CI server on premises would probably pay itself in less than three months.
Startups are usually resource-rich and time-poor, and don't want to allocate employee time to infrastructure unless it's core to their business, so arguments for efficient spending there are easily overlooked.
We used to have self-hosted runners on Hetzner dedicated servers, and we migrated to UbiCloud ~6 months ago.
Unless you can optimise your CI runtime to be very balanced and fast, you're going to have a lot of jobs queued during working hours ... although it costs more per minute, the fact that CI jobs are bursty means it works out not too much costlier but with less queue time.
Oh good point. Hadn't thought of the bursty nature of the load.
Sounds like you need to share infra with an org of similar needs in a vastly different timezone.
I guess that's what SaaS CI solutions are in a way. :)
Hey folks - Greg here, I do product at Blacksmith.
I want to say upfront - we've never pursued these invoices. If someone feels they didn't get value from the service, we've eaten that cost and always will.
There's a bit of an implicit policy decision on our side here that we did a bad job of communicating - I want to clarify that, and then talk about how we can fix it.
First, we wanted to let customers start using Blacksmith without a credit card. Very few infra startups do this today - CC validation is great for anti-abuse - but doing so has let us support a much greater number of free users.
Many of these free users have turned into full OSS sponsorships (most recently ccusage, and before that, OpenClaw) or large paying accounts, and we haven't wanted to cut those users off from trying us. It's a real pain point to find a credit card to put down for your company or OSS project's CI spend before you've even tried the service.
Second, not having a credit card on file means we don't actually know when credit card-less users intend to continue past their free trial.
Shutting down users CI workloads entirely seemed harsh, especially because doing so would fail builds and require a code change to resolve. If users could start without a credit card, we weren't going to then hold their runners hostage. Instead, we decided to just eat the cost for the small number of users who either abused our services or did not actually mean to use the service.
This worked, mostly - though every month we have gotten a number of support cases with users confused about their invoice. If they didn't intend to use the service past the free tier, we've voided out the invoice, and often given credits against a future bill if they intended to use the service, but were surprised by the behavior.
We have a lot to improve about our billing mechanics - but because our retention rate for these users has been so high, we have assumed great support could catch and resolve the ambiguity.
That said, there's two changes we can make now:
1. we clearly missed the mark on supporting this specific case - we should have offered to void this bill entirely given the surprise factor here.
2. We're prioritizing up making progress on a Wallet implementation that will let folks choose to suspend their runners rather than let them continue to run after they use up their free tier.
We also just launched a new billing/metrics view so users have better visibility into their free tier and Blacksmith usage.
I'm sorry for the bad taste this has left in everyone's mouth - I'll be hanging out here and on greg [at] blacksmith [dot] sh if you want to talk about your account specifically.
As a consumer, I can appreciate your focus on support. I get a monthly $0.00 AWS bill on an account I've lost access to. However, because I have no outstanding balance, support is unable to assist me (is their claim). So I enjoy one monthly spam email from them, which I neurotically read in case one month the balance for whatever reason exceeds 0. Still less stressful than trying to resolve the issue with them.
> I want to say upfront - we've never pursued these invoices.
At first glance that sounds admirable, but the flipside is that it implies Blacksmith knows they're being shady: if you know you're not going to pursue it, why did you invoice the customer in the first place? This sounds less like you're forgiving a charity case and more like you're waiving an invoice because the customer identified it as unethical to begin with.
I appreciate you stepping in and clarifying. Thank you.
But honestly, this feels like damage control. This is (to me) clearly an innovation on a dark pattern that is basically just "accepted practice" nowadays, namely the "subscribe by default; make it hard to opt-out of said subscription at signup time".
That's why I think people are upset about this. By not taking a credit card, you made it feel like it wasn't an implementation of that dark pattern (yay!) -- however, secretly, it was!
If the intent is to be customer friendly, it's so perfectly clear to me what the answer is. OPT-IN. In other words, a checkbox:
"By default, when your free credits are consumed, all of your runners will be de-provisioned. Instead, if you would prefer that your runners continue working, check this box and we will invoice you for usage in excess of your free credits."
But honestly, you did more than I expect of most service providers today. You sent an email; you actually told them how many credits were left in the free tier. AWS, for example, can only give me an _estimate_ of how much cost I've accrued. They can make no promises about the rate at which I'm expected to accrue new costs. And, unless I've taken great care (by, say, terraforming every resource in a given AWS account), "turning off my cost accruing services" is not a simple matter. If I understood the article correctly, there was, at least, a "single action" they could have taken to immediately stop accruing costs.
I'm not so sure that Blacksmith is making the right choice around how to handle this signup flow, but I have very little sympathy for the person in this post. $1000 in CI is a lot of usage.
The main issue is that you deceptively communicated what happens at the end of the free trial. You decided to use the wording "disruption", which implies that workloads will be stopped, which is the opposite of what happened. This is deceptive and predatory. I will never use your software and after this happened I am AMAZED that anybody else (particularly OP, but also anybody who read this post) will continue to trust you with their workloads. The way I look at this is: if you're willing to do deceptive and predatory billing, why wouldn't you be also willing to sell access to intelligence agencies or some other "grey area" party who wants to do supply chain hacks?
Hey -
First off, fair point on the free tier email. This fits in the same theme of clarity vs. trust for me: if we say disruption, but then leave your account enabled, that still feels deceptive even if it seems generous to us. We will fix.
On your last point - I just want to note that this isn't a money-maker for us. Runner bills can be large (as they were here!), and we explicitly wanted to avoid actual charges to the user in favor of not collecting for a good amount of usage. Again, there's a clarity/trust tension there that you're right to call out, but the end goal is that more people can use cheaper, faster runners.
We earn that trust today by being extremely easy to use and delivering on what we promise, and we need to add clarity to that list. Thanks for the feedback.
This whole incident supplied them a lot more publicity. I had never heard of them before today. Since today, I'm more likely to use them, not less, because now I know they exist. I couldn't be less likely than zero.
Awful, but true.
Years ago I got my first internet enabled mobile, and the carrier advertised it as having "300 free minutes" as part of the trial period, which is great. So I used 297 minutes of internet services in the first month, but aha actually the minutes only referred to telephony, and I was stung with a ~12,000 dollar overdue mobile internet invoice (360 dollars per megabyte or something equally ludicrous)
They got done in by a massive class action, that I was tangentially a beneficiary of, not because of the minutes claim, which was standard practice, but because they had failed to provide anyone with the cost of the data.
I think I paid them 300 bucks or something in the end. After further letting a 600 dollar agreement go to collections and settling with collections for 50%.
Many many years ago my girlfriend at the time was on holiday with her mother so I'd added a "Roaming Bundle" to her phone.
While she was away I got a call from our phone provider that she'd racked up £1700 of roaming charges. "But don't worry about it!" said the customer service person, "Just stop your direct debit for this month just in case and we will call you when the bill is generated."
A couple of weeks later, they rang back, confirmed that the final amount was about two grand, and so I said "hang on, there was supposed to be a message when the bundle was halfway used and a message when the bundle was nearly used up - what happened to that?"
"Oh I don't know why you'd have been told that, we've never done that! But you're right, that's what they said, we've already listened to the call. What you *should* have been offered was this *other* bundle which would have cost £150, she's used about 75% of the data that would provide, plus the £25 you originally paid for the first bundle, call it 75 quid then? I'll just take the card number..."
And that, gentle reader, is why I'm still a customer.
Data was counted as minutes back then for you? I only remember kB and MB costs...
In the early 2G GSM days, you would connect via "CSD" (Circuit-Switched Data[0]) which essentially emulated a dial-up connection (except it was digital the whole way) and that was billed per minute. Using HSCSD you could bundle multiple CSD channels for higher bandwidth but the minute charges were per channel so the cost added up.
By the time phones got built-in WAP browsers, GPRS had arrived which was billed per byte, but CSD was still supported (I remember using it once or twice in a certain building at uni where GPRS never worked for some reason)
If you were used to dial-up internet, (IE, where your computer was plugged into a phone line and made a phone call to your internet service provider,) that's a reasonable assumption to make. Some early services billed by the minute, too.
It was one thing to use the line to call an ISP, which can be a company of its own, and another to use their Internet access on that call.
Very early on, mobile data was essentially an emulation of an analogue modem call, at blazing fast rates like 19200baud or less, held over the same channels as voice calls with a bit of extra jiggery-pokery, and were charged by the minutes just like voice calls. Sometimes they were charged as normal voice calls, sometimes you ended up being charged extra (or not being charged extra as such, but with data calls not being counted in any pre-paid minute allowance your plan had). It was vitally important to check this before making modem calls, as some providers charged data calls in the same band as premium rate voice calls, meaning that just picking up a few text emails could be quite expensive.
[cue misty eyed memories of trying to get my Psion 5mx to connect to the Internet by infra-red serial link to my phone, to try get updated connection information for my next train as the one I was on was going to be too late to make the planned connection…]
Last century most people were only sometimes using the Internet. They'd "Go online", if you're young enough not to recognise that noise in Blue Prince when you use the network, that's a "Dial up modem" which is how a typical person would "Go online" in the mid-1990s. So in this regime counting minutes makes some sense.
I got a preview of the modern world from about 1996 because my first shared student house (shout out to any Hitchers reading) had a single dial-up modem set to always connect to a free-to-use University modem and then used IP Masquerading (the ancestor of today's NATs) so that all of our computers could share this tiny connection.
So by the time of my 21st birthday, I was "always on" in the same sense that you'd always be today, except with much, much lower bandwidth and what I can tell you is that this, not the bandwidth is what makes the difference.
When you're "always on" the reflexive answer to "Wait, where do Porcupines live?" is to look online. It's 1996 so Wikipedia doesn't exist yet. Google doesn't exist yet. But Tim's crap hypermedia system (the "World Wide Web") exists and so you just need to know where to look to find information about porcupines.
I didn't watch videos in 1996 because it'd take hours to receive a short low resolution video, even reading web comics was quite an undertaking, I remember downloading all of Bruno (at the time) https://en.wikipedia.org/wiki/Bruno_(webcomic) over night so I could read it. But the fact you don't have to explicitly "go online" makes a huge qualitative difference even though the bandwidth is tiny.
Things I learned today: the very first mobile (cellular) data protocol was indeed dial-up over the GSM voice line: namely the CSD. What I had in mind - GPRS - came only later... so yes, even on mobile it was initially billed by minute. Wow.
Same here and I started uni in 2003. At home it was dial up and you had to unspool the wire across the house from the phone plug to the computer.
At uni it was fast. So you could actually load images by default.
> if you're young enough not to recognise that noise in Blue Prince when you use the network, that's a "Dial up modem" which is how a typical person would "Go online" in the mid-1990s
I have some lovely high-quality video of dialup modems both dialling out and dialling into with good audio, which I'll post up at some point.
We still use them at work, at least until BT finally cut the copper services off at the end of the year.
The similarity between mobile internet (CSD) and dial-up wasn't really obvious from the users' perspective because they weren't explicitly making a call to access the internet. The session was established transparently when data had to be transferred, and the time this took was charged as minutes.
Operators always dreamed of a world built on circuit-switched networks that they fully control instead of packet-switched IP networks where anyone can take part and operators are just a carrier. So the big operators started the mobile internet era with the telephony model.
Circuit switching has latency and QoS advantages, but it's definitely not worth the cost for most use cases.
Interestingly, TDM carriers have a latency of approximately one bit per hop and jitter less than that, while any kind of packetized voice such as VoIP has a latency of at least one packet worth of audio, plus one to several packet transmission times per hop, and jitter of several packet transmission times, and packet drop on top of all that. We've actually made quality worse in the name of fitting more services down the same wire, which isn't necessarily bad but it's something to think about.
Internet access used to be billed by the minute back in the Prodigy/Compuserve/AOL days, and early phone plans mirrored that for ease of marketing.
Other way around. The phone plans were billed by the minute. Then came the Internet. In is first generation, Internet was essentially just a long distance call through a modem. Hence it was billed - like the call - by the minute.
Dedicated Internet wires came much later, and then the dedicated phone lines were dropped as voip was better quality and cheaper compared to the dedicated lines.
While the phone still had a dedicated line it didn't actually need a power connection, as the power through the phone wire was sufficient.
Online services most assuredly billed by the minute. He'll, AOL had a huge marketing campaign offering "free" minutes for new customers. You might also have charges from the phone company but those were independent of the online service charges. It wasn't until the late 90s the major online services went to flat rate "unlimited" plans.
I'm thoroughly confused as to how what you're saying differs from my previous post, but I believe you just misunderstand that by "early phone plans" in my post's and the parent post's context, I meant phone plans for Internet access ("early" being the feature phone era).
This is why I'm pretty sympathetic to hard caps by default
I love hard caps, and am tired of cloud services not even offering those. May make sense for large companies. Makes no sense for hobbyists and small companies. But maybe that's the point?
For consumer products, definitely. Any unreasonably large charge (with a comfortable margin, like 10x) should be waived by default, unless the client specifically requested the limit be removed.
That makes me think of bank overdraft fees.
They are another kind of "soft" cap, pitched as an automatic convenience to the customer... But in practice they are too-often deceptive and harmful.
https://www.library.hbs.edu/working-knowledge/are-banks-the-...
When one of my sisters got her first paycheck from her first real job, she didn't knew much about banking, and thought that the overdraft limit on the ATM was her actual salary, so she just withdrew the whole thing. I think it took over a year, maybe two, to pay it all back.
i always set my overdraft limit to zero. i never want to use that by accident.
In the US this always seems to be called "overdraft protection", as in "do you want overdraft protection on your account?"
I can never remember if overdraft protection means "we won't let your account overdraft", or "we will let your account overdraft"
It is an incredibly confusing name to me
I could look up the answer now, but I know I will just forget it again. I must have looked it up and then forgotten at least 10 separate times over the years
Normally means that if your checking accout balance hits zero, they will pull from your savings account rather than returning a check presented for payment. They may (probably will) still charge you a fee when this happens, but it might be less than an overdraft.
Or, it might be a case where they grant you a short-term personal loan to cover the overdraft (up to some limit) rather than return checks. Again there will be fees and interest on this.
In either case, since they are not rejecting payments, you avoid getting hit with fees from whomever you wrote the check to. So your only fees/penalties are paid to the bank.
Of course the best answer is just keep better track of your checking account balance.
It probably depends on the bank, but setting one's overdraft limit to zero doesn't necessarily help and may actually make things worse: some payments may still go through even if you don't have the funds, putting you in unauthorised overdraft, which tends to have a high daily fee.
That is insane. What's the point of overdraft limit if it doesn't limit your overdraft?
First time? You can choose between the $20 convenience fee or the $50 no-fee fee.
This reminds me of the business practices of the Austrian NIC. Usually domain names expire if you don't renew them. In Austria, unless you explicitly cancel the domain name by fax, they just roll the registration over to the next year and then send you to collections[1] if you don't pay up.
There's no rule that domain names expire unless you renew them, at least for ccTLDs. It's just a convention. Conventions lead to assumptions, and assumptions can be used to scam people.
In general there's two types of businesses: businesses where you pre-pay (e.g. McDonalds), and businesses where you post-pay (e.g. a sit-down restaurant). If you take a conventionally pre-pay service and apply post-pay pricing to it, you have yourself a perfect scam.
[1] https://www.reddit.com/r/sysadmin/comments/1bnjus/the_austri...
This would not surprise me with a German service provider, although it got much better in the last decade.
[dead]
There are, in fact, absolutely no US-imposed rules about how a country manages its ccTLD like there are on gTLDs (words).
By "US-imposed" you presumably mean ICANN-imposed.
Yes? ICANN is part of the US, and has been strong-armed by the US government into imposing restrictive rules on gTLDs even though they appear like they're global.
Any actual evidence that the US government has control over ICANN? I know they used to have an agreement that gave the US some oversight, but as far as I know that ended a decade ago.
They do that or they did that? (you are linking to a decade old post)
Seems like they still do:
https://www.nic.at/en/how-at-works/domain-holder#id105
> I received a letter from the debt collection agency. What can I do?
> If a domain hasn't been paid for despite several payment reminders from nic.at, the domain shall be locked and the open claim handed over to our debt collection agency. As a result, the invoice must be paid directly to the debt collection agency. Please contact our debt collection agency for more information
Like TFA it's hard to tell if they genuinely believe that they are helping their customers by not discontinuing their service, or if it's a scam. I suspect a mixture of both.
It's both. It's normal business practice in the German-speaking countries (a.k.a. DACH) that would be considered a scam anywhere else.
Basically, in most countries paying money is something that requires continuous enthusiastic consent - if you don't pay, that's the business's problem and they should stop serving you, and they may only recover payment for goods they've already given you and not received payment for. But in DACH, it only requires technical consent - if you signed something saying you'll give them money, then you have to give them that money, and you cannot rescind your obligation to pay, except as provided in the contract or an overriding law.
You went to Austria and did Austrian business with an Austrian company, you should be aware that Austrian rules and norms apply. ccTLDs are not generic, every country is free to apply any rules on their ccTLD!
This creates some very uncomfortable cultural clashes sometimes. I remember way back when I was signed up for a service provided by Ms. They did not, as far as I could tell, provide a way to cancel the service, other than to stop paying. Since I had provided them with automatic payments, my bank did not allow me to simply stop paying their invoices, I had to cancel with the provider. I talked to my banks customer service, they refused to help. So I cancelled my account with that bank and moved all the money to a different bank.
Rather more complicated to cancel that service than I would've wished, but hey, I got a better banking service out of the deal, so it's not all bad I guess
Ms = Microsoft?
Being unable to tell how to cancel something is a different thing. You can contact their support. If they don't have support, or the support refuses, then you contact your bank, with evidence, because it's now an unauthorized payment.
This is separate from the cultural norm that you can only cancel with a certain notice period or by fax.