After working on my social app in 2023, I wasn’t really sure where to take my career from there. After some freelance projects, I wasn’t sure what to work on next. I was talking to a previous co-founder, the one from the messenger app, and he said that he was working with somebody in the States who had an idea for a business, and we might be a good fit.
The idea
He had a lot of experience in B2B sales and lead generation, and we collaborated on a tool that was supposed to rebuild the entire LinkedIn database in order to do data mining on top of it and determine who might be a good candidate to reach out to based on their activity.
The way it was going to work was that we were going to start with a simple search for some keywords, finding people that come up in LinkedIn search, and then jumping maybe four or five connections down the graph based on what they’ve posted about, what posts they’ve interacted with, and which other people have made those posts or interacted with them. That way, we would grow a network of people and, at the same time, also grow a network of activity per person.
That way, you can not just find somebody who’s working in your industry, but really see what people are talking about and then react to those things directly in a DM that’s not quite as cold as if you’re just trying to sell somebody something out of the blue.
The goal was to not optimize for a sale in a cold outreach, but to optimize for a response with a proper human-sounding message responding to something that they’ve interacted with recently on LinkedIn and starting a conversation from there that hopefully has a high probability of getting a response, because we’re not selling anything. We’re just talking.
I think the idea is great, and if nobody’s done this so far, I’m sure it’s going to happen a lot more. I still feel like outreach hasn’t really improved much since the dawn of AI, and I think there is a huge opportunity here left on the table.
The crawling problem
I am, however, pretty sure I’m not going to continue working in this space ever, because boy was it frustrating to get access to all of this data and try to get around LinkedIn’s bot detection stuff.
I think maybe, as always, I was over-optimizing for cost and trying to get away with no proxy usage, because residential IP proxies are pretty expensive for bandwidth. So my idea was that the only way to make this work from the business fundamentals was going to be using a way where Amazon data center IPs don’t result in a ban. I ran very extensive testing on that.
The core idea was going to be to use AWS Lambda to get a huge pool of rotating IP addresses in order to hopefully not make it look like all of these accounts come from the same server. If you know anything about crawling and/or LinkedIn’s bot detection, though, you already know how this story is going to end, because they are doing a pretty good job of obfuscating exactly what the action was that caused the ban or limit, but they do take very strong action.
So I ran some preliminary tests after creating maybe 100 accounts and aging them. I was trying to put them into the system that I’d built to see if they got banned, and after making an initial comment or post or even just a hat tip to somebody’s activity, everything seemed to be going well.
But it turns out that the system did flag this as bot activity, and it just took a while before it knocked out the account completely and banned it.
So I think the lesson learned here is that the systems that detect all of these bots are way more sophisticated than you think. And if you think you found a clever hack to get away with solving bot detection without buying expensive residential proxy bandwidth from a well-known provider, then you’re probably just wrong and overconfident.
The risk
Another issue is that most of these crawlers do ban LinkedIn traffic, presumably because there were previous lawsuits and litigation not allowing them to access LinkedIn anymore, which is fair. So I think LinkedIn in particular might just be one of those targets that’s only worth hitting if you know that you have a shot at a goal, meaning that you need to have an existing business model, existing customers, and a high likelihood of turning a profit. Otherwise, it might just not be worth trying all of this and wasting time and spinning your wheels trying to fight this giant.
I’m sure it’s possible. I know people that do this, but for me personally, the learning here was definitely that crawling is just a space in which the issues often manifest way down the road, and it’s not really a fun place to prototype in, because what you’ll learn one day might not be valuable anymore tomorrow.
I also feel like the morality of doing this isn’t super straightforward. I like to solve problems, and I like to build things people want. This would just be building something that marketers want, and that’s not really a target audience that I find myself resonating with quite as much as, say, the average New Yorker taking the subway.
The collaboration lesson
So all this to say, I think it was an okay idea. I think I was overly optimistic.
Ultimately, it didn’t work out because the person who my previous co-founder introduced me to was job hunting while also working on this project. He didn’t put in that much time to begin with, but once he landed a job, he was just completely out.
It was maybe three months of my life working on this with 60% of my time, and I don’t think it was a waste of time, but I think some of the issues here could have been seen sooner, and I could have been a bit more efficient about it. Especially the fact that I was trying to co-found something with somebody who was actively job hunting was a pretty big red flag.
I think that again, optimism about collaborations just comes to bite me again and again. He was, of course, optimistic about his time and said something like, “Oh, even if I find a job, I’m going to be working on this part-time still,” but that didn’t manifest. And I was optimistic that we would get something off the ground before he could even find a job, and it was all going to be okay.
Maybe optimism isn’t a bad thing, but I think that especially when it comes to looking for and designing long-term collaborations with people, it does sometimes pay to be pessimistic about reasons why it might not work out, and then try to have an honest conversation about what the odds really are and what might happen in the best-case scenario.
Human relationships take a lot of time, and more than anything else in entrepreneurship, whether you work with the right person or not is your highest indicator of success. So trying to optimize the speed of finding whether somebody is a good fit or not might be a good lesson here.