211 points by vancecookcobxin about 12 hours ago
65 Comments
hbarka about 12 hours ago
.
operatingthetan about 11 hours ago
I don't understand why that youtuber was acting like spy satellites going over was such a big deal, they are going over the entire planet, all the time.

edit: no idea why they deleted the comment but they linked to this video https://www.youtube.com/watch?v=0p8o7AeHDzg

kjs3 about 11 hours ago
Claude told him it was a big deal. Why would he question Claude.
whattheheckheck about 11 hours ago
Yoooo this is amazing... can you add rss feeds like feeder.co aggregating subreddits and groundnews articles embedded in here too?

And add chronological feeds of govtrack.us along with all politicians social media feeds

btbuildem about 11 hours ago
Lol please at least clean up the markdown diagram -- claude has a real hard time aligning the borders in ascii art for some reason.
vancecookcobxin about 11 hours ago
Thank you for the heads up! Will do.
erichocean about 11 hours ago
Yup, I had Claude write a tool to auto-fix those diagrams. :D
mentalgear about 11 hours ago
dont give these OSINT quality signals away ... that's one of the indicators that allow you on first scan to id (potentially) low quality content. Ie: fully llm gen; the author doesnt look over the docs or doesnt care for 'details'.
totetsu about 2 hours ago
“The first Matrix I designed was quite naturally perfect, it was a work of art, flawless, sublime; a triumph equaled only by its monumental failure. The inevitability of its doom is apparent to me now as a consequence of the imperfection inherent in every human being. Thus I redesigned it, based on your history, to more accurately reflect the varying grotesqueries of your nature. However, I was again frustrated by failure. ”
afatparakeet about 11 hours ago
Optimizing some of that geojson into realtime tiles is a really fun and engaging project.

Have you seen these projects?

https://github.com/protomaps/PMTiles

https://github.com/maplibre/martin

vancecookcobxin about 8 hours ago
They are definitely on the horizon! I am a HUGE fan of both of those projects and they are definitely on the roadmap for the architecture...

Right now, ShadowBroker is really optimized for 'blinking blip' real-time radar tracking (streaming the raw GeoJSON payload from the FastAPI backend directly to MapLibre every 60s), so we get as close to as smooth 60fps entity animations across the map.

Moving to something like Martin would be incredible for handling EVEN MORE entities if we start archiving historical flight and AIS data into a proper PostGIS database, but the trade-off of having to invalidate the vector tile cache every few seconds for live-moving targets makes it a bit overkill right now....

afatparakeet about 4 hours ago
Yeah less ideal for the realtime data but could be useful for lightening the load of certain more static layers.

Great project, will be contributing!

vancecookcobxin about 3 hours ago
Glad to have you aboard!
laborcontract about 11 hours ago
I've seen so many of these in the last week alone.

I need a realtime OSINT dashboard for OSINT dashboards.

operatingthetan about 11 hours ago
Which is the best one so far?
laborcontract about 11 hours ago
I'm going to have to create an Awesome Best OSINT dashboards github repo to answer that.
skinnymuch about 8 hours ago
Reminds me of all the Covid data trackers in mid 2020
nonameiguess about 6 hours ago
It's pretty interesting to see. My very first real software job was working on ground processing algorithms for the US Navy's Maritime Domain Awareness system, which is the "real" version of something like this that actually gives centimeter scale live activity detections of basically the entire world. The engineering effort that goes into something like that is immense. Bush announced in like 2004 or something and we didn't go into full operational capability until 2015. Thousands of developers across intel, military, commercial contractors, for over a decade, inventing and launching new sensor platforms, along with build outs of the data centers to collect, process, store, and make sense of all this.

I wish these weekend warriors would work on a project like that someday, to see what capabilities truly take. You want to know what's happening in the world, you need to place physical sensors out there, deal with the fact that your own signals are being jammed and blocked, the things you're trying to see are also trying to hide and disguise themselves.

The attention to detail is something I've never seen replicated outside. Every time we changed or put out a new algorithm, we had to process old data with it and explain to analysts and scientists every single pixel that changed in the end product and why.

vancecookcobxin about 6 hours ago
I get it! Unfortunately, you need a security clearance or a really fat wallet to get that kind of data. OSINT is a different thing.
the_real_cher about 5 hours ago
One guy vs the DoD

apples and oranges

rustyhancock about 11 hours ago
There's no data when I tried it on a windows 11 PC. It seemed to install all deps front end is served but dossier says intel unavailable.

No planes etc.

No helpful output in the command window.

Seems fun but doesn't seem to be working.

spzb about 11 hours ago
Same on a Mac
DetroitThrow about 11 hours ago
Yeah this doesn't work on Mac either. This is just broken and nonfunctioning.
vancecookcobxin about 11 hours ago
Apparently, I had a bunch of front end developmental scripts that were calling the Windows version of python. Working on it now.
vancecookcobxin about 11 hours ago
Ah, that's my fault for not making the error handling clearer in the UI. If the map is blank, it usually means the backend is missing the .env file with the free API keys (AISSTREAM_API_KEY and N2YO_API_KEY), so it's silently failing to fetch the streams.

Did the terminal throw any Python FastAPI errors, or did it just serve the Next.js frontend? I'm going to push an update later today to show a prominent "Backend Disconnected / Missing API Keys" warning on the UI so it doesn't just look dead. Thanks for testing it!

rustyhancock about 10 hours ago
Looks like I had to use python-3.11 and install a few dependencies.
AH4oFVbPT4f8 about 7 hours ago
On the topic of API Keys, for Opensky it's OPENSKY_CLIENT_ID and OPENSKY_CLIENT_SECRET, the readme has OPENSKY_USERNAME and OPENSKY_PASSWORD
AH4oFVbPT4f8 about 7 hours ago
Change the requirements.txt in the backend to the following

fastapi==0.103.1

uvicorn==0.23.2

yfinance>=0.2.40

feedparser==6.0.10

legacy-cgi==2.6.1

requests==2.31.0

apscheduler==3.10.3

pydantic==2.11.0

pydantic-settings==2.8.0

playwright>=1.58.0

beautifulsoup4>=4.12.0

sgp4>=2.22

cachetools>=5.3.0

cloudscraper>=1.2.71

reverse_geocoder>=1.5.1

lxml>=5.0

python-dotenv>=1.0

and be on python 3.13 and it should get you up and running

ionwake about 11 hours ago
Really cool thanks for sharing. What are the API costs like if i ran this for a couple hours a day for a month? Is it affordable?
vancecookcobxin about 10 hours ago
Its all free baby lol
serf about 11 hours ago
cool idea.

first llm to stop using those damn colors for every single transparent modal in existence is going to be a big step forward.

vavkamil about 10 hours ago
You leaked `./frontend/.env.local` & `./backend/.env` inside `ShadowBroker_v0.1.zip` in the first commit.
DetroitThrow about 10 hours ago
the real OSINT is always in the comments
tfghhjh about 8 hours ago
thats why its called osint

everything is open source

hettygreen about 10 hours ago
This looks really cool..

Let me ask a dumb question. Can this be run on a public server (I use dreamhost) with a web interface for others to see? Or is this strictly something that gets run on a local computer?

vancecookcobxin about 10 hours ago
Well, I have to make some modifications, but that isn't recommended right now because I have a settings option with the API key right there for the free world to see, lol. I will work on making a version for hosting it, though.

You can throw it on a server and run it for you to see (or anyone else if you trust people or dont care about losing your free API keys) It's just a standard Next.js and FastAPI stack, and there are Dockerfiles in the repo so it should be pretty straightforward to spin up on a cheap VPS (like a DigitalOcean droplet or Hetzner).

Honestly, if you just want to show it off to a few people, running it locally and exposing it with a Cloudflare Tunnel or Ngrok is probably the path of least resistance.

I WILL work on having a version to host it where users have to bring their own keys to see it in the future though

silverstream about 7 hours ago
Cloudflare Tunnel is solid for quick demos. One thing though — if you're planning the "bring your own keys" version, don't just throw them in a settings page. I went down that road and ended up with keys sitting in localStorage where any XSS could grab them. What worked better for me was having the backend hold the keys and issuing short-lived session tokens to the frontend. More moving parts but way less surface area if something goes wrong.
vancecookcobxin about 6 hours ago
Stellar advice! I will totally keep that in mind. Thanks!
Karrot_Kream about 7 hours ago
If you want to host for friends/trusted devices, you can put it on a Tailscale or Zerotier style network and just let trusted devices access the server wrt to the OP's point about open secrets. Or you could probably make a PR to load the settings from somewhere else.
garyfirestorm about 10 hours ago
> Do not use this tool for any operational, military, or intelligence purpose.

How long before we see this UI in some Iran related news story

vancecookcobxin about 7 hours ago
I REALLY, REALLY dont want it used for that type of stuff.
CountGeek about 10 hours ago
rationalist about 7 hours ago
Risky click. (It's okay.)
born-jre about 5 hours ago
i was building sth like this

https://github.com/blue-monads/potato-apps/tree/master/cimpl...

i should finish but have not have time

efromvt about 10 hours ago
I'd be interested in just the data layer of this being extractable - will poke around at that. (frontend is fun, though!).
coolius about 9 hours ago
i wish someone could deploy this somewhere so we can try it out without having to build it first
crawfordcomeaux about 9 hours ago
I'm excited to see tooling of this nature and scope. Looking forward to seeing similar tooling oriented around all human needs so we can start tracking the meeting of needs to better meet needs, particularly in ways that don't require money.
jll29 about 6 hours ago
Thanks for opening this up.

As was already said in one of the reference videos, it's impressive what one person can do.

But the next step is to define an architecture where authors can defined/implement plug-ins with particular modular capabilities instead of one big monolith. For example, instead of front-end (GUI) and back-end (feeds), there ought to be a middle layer that models some of the domain logic (events: surces, filters, sinks; stories/time lines etc.).

I would like to see a plug-in for EMM (European Media Monitor) integrated, for instance ( https://emm.newsbrief.eu/NewsBrief/alertedition/en/ECnews.ht... ).

amelius about 8 hours ago
Does it show locations of datacenters?
driverdan about 8 hours ago
What's with so many people creating new accounts to promote LLM generated projects? Are they people who don't care about HN and just trying to self promote? Existing users creating new accounts? Lurkers?
alephnerd about 7 hours ago
How is this AI slop? It seems functional and actually reminds me of a couple alphas I saw of similar threat intel products 10-15 years ago.

Of course it's commoditized and a dime-a-dozen today, but if this is what HN terms as "AI slop" then apparently human SWEs weren't that much better.

driverdan about 7 hours ago
I never said AI slop.
alephnerd about 7 hours ago
Ah! I misinterpreted your comment then!
beepbooptheory about 7 hours ago
It's a bummer because sometimes the headline seems cool, but its always generated blah blah recently. I don't think I've seen a non-AI readme on here in months..

Everyone has their own hueristic, but if it took someone 6 hours or whatever to make some whole big app, my confidence that they will continue to maintain or care about it even next week is pretty much zero... How could they? They've already made three other apps in that time!

I don't care if the code is perfect, all this stuff just has the feel of plastic cutlery, if that makes sense.

polynomial about 7 hours ago
Plastic cutlery is a dead-on perfect analogy.
gregjw about 6 hours ago
Plastic cutlery, thats great.
chid about 8 hours ago
Did I see this on X first?
pugworthy about 6 hours ago
I was hoping for something like the old Henchman's Helper site, which went offline around 2016-17.

Archive version...

https://web.archive.org/web/20120112012912/http://henchmansh...

anigbrowl about 3 hours ago
I’ll admit I leaned way too hard into the "movie hacker" aesthetic for the UI

Nothing wrong with that. Beats a boring corporate dashboard any day. Video game and similar interfaces work for a reason.

david_shi about 2 hours ago
not knocking this specific implementation in any way, but it's crazy that live OSINT dashboards are now the demo project of choice vs. todo apps
raised_hand about 1 hour ago
Is this hosted anywhere?
4mitkumar about 1 hour ago
Try this https://www.worldmonitor.app/ for a hosted version of this...from a different dev but very, very close.
fittingopposite 7 minutes ago
Website ist down..
4mitkumar about 1 hour ago
Very cool! Although, the concept, the feeds, the design and everything reminds me of https://www.worldmonitor.app/ - also live and deployed btw, if you want to check out the interface.
poemxo 7 minutes ago
Why the name Shadowbroker? It sounds a lot like the Shadow Brokers which is the hacker group that stole and published some NSA hacking tools.

Then again they were named after a video game character so it's probably fair.