It is a neat example of how an org can structure and manage multiple projects and services in a single git repository. They've use Leiningen to achieve their objective.
> The Common Metadata Repository (CMR) is an earth science metadata repository for NASA EOSDIS data. The CMR Search API provides access to this metadata.
> Building and Running the CMR
> The CMR is a system consisting of many services. The services can run individually or in a single process. Running in a single process makes local development easier because it avoids having to start many different processes. The sections below contain instructions for running the CMR as a single process or as many processes.
The ongoing discussion for Biff [1] prompted me to re-share my post because I'd like more people to understand this "other way". Outside Clojureville, it is not obvious most of these Clojure "frameworks" are not monoliths.
The consummate Clojurist's default (and very normal-feeling way) to build a web application (or any application for that matter) is to roll their own web stack from production-grade libraries.
Of course, this state of affairs is a double-edged sword, just like is true for traditional web frameworks. In my post, I try to go into the whys and the wherefores, building upward from first principles.
I'd say the nice thing with the Clojure way of building your own stack is it becomes quite easy to swap parts out. On a previous project we swapped out our web server three different times with minimal changes (jetty -> aleph -> httpkit) as for the most part they all shared the same interface.
After a while you get good at seeing where you want things to be configurable and where you don't. It also gives you the confidence to roll your own micro stack/framework which means you are not dependent on third party aggregates to adopt new features.
Thank you for the kind appreciation. Made my day :)
Yes; next to the sheer stability of parts, their fungibility is a business-critical feature of the Clojure ecosystem. Of course said fungibility does not magically manifest. However the effort to get there is "not much", I'd say. The use of "system" libraries, with some well-reasoned module design brings it pretty close to magic.
As in the post, a fungible, production-grade part can be just a multimethod (e.g. the router in the post). Why? Because "production" comes in all sizes. A small SaaS with a few hundred customers may chug along happily with a bunch of functions.
Hm, I'm saying a given function can be production grade. I'm not saying all functions are production-grade. Also, I'm saying production comes in all sizes. If your micro-SaaS app gives you a livelihood, that's hella production --- real skin in game, real stakes in ground.
To analogise further...
- HackerNews is a "production" system, you would agree. Back in 2015, it was still true. Are flat files a "production grade" primary data storage choice? [1]
- Suppose your production service transacts a million requests an hour (say it is a short-link maker). Further, let's say it has only a handful of API endpoints. Do you really need a whole routing library for that, if a single multimethod does exactly what you need, correctly?
Right. And what I’m saying is that if “production grade” means all of these things, then it’s not a very meaningful qualifier.
My personal site is statically generated — it’s just a bunch of HTML files and some CSS. Do you not think it would be a bit pretentious of me to describe that as “production grade”?
If “production grade” simply means fit for purpose, then given the GP commenter’s reason for their initial web server swap, wouldn’t you say by definition that jetty is not production grade, since it doesn’t (didn’t?) support SSE?
I agree with you... "production grade" is in fact a meaningless term without saying what grade one's production needs to be.
A static site that serves is most definitely a production system. Perhaps one that could scale in traffic almost without end.
evalapply.org gets (to my continued amazement) 20K+ unique visitors a month when it's business as usual. On a busy HN day, it's easily that much in hours. I don't have to think about "scaling problems". I don't have one.
> On a previous project we swapped out our web server three different times
Why is this desirable?
On all my projects over the past 10 years, I’ve swapped out the web server exactly zero times because the one I have works just fine. The parent comment describes these components as “production grade”, but then if that’s the case, what could be the reason for swapping them out other than self-indulgence?
It's not. But, the fact that we could and it was straight forward still amazes me.
In our case the first time we needed to, we needed SSE and at the time there was no ring-jetty async interface (it's a long time ago so I'm forgetting the details) so we moved to aleph. Much later, we wanted to try out http-kit (self-indulgence) as we were operating behind a proxy anyway for performance reasons and it made a significant difference.
If we'd just started with http-kit that would have been fine. I guess it comes down to what features you need.
Also I 100% agree it's something ideally you would want to avoid a in the case of databases for example so much performance is left on the table because for some reason we want to be able to swap between SQLite, postgres and mySQL. Which in practice you never want to do.
As the former maintainer of Aleph, I'm very surprised you ran into a situation where http-kit was faster. Or do you mean it was just preferable to develop for?
Generally, yes one would not want to swap out their web server willy-nilly...
Yet, this is one of those "YAGNI in 99% of your use cases", but when that 1% use case arises, a server swap would be far more desirable than a whole framework shift.
So while self-indulgence can certainly be a motive (and why not? as long as everyone's having a good time), may I offer a few more charitable reasons for this:
- programming API ergonomics
- performance
- application runtime model (servlets -> embedded server)
- security model
- application server features (websockets, comet?)
- binary size
- server configuration niceties
etc...
That said, a developer only has flexibility if it is built in from the get go.
A counterfactual would be to consider the set of developers who have had to put in ugly hacks because they can't just rip the web server out of the framework of choice they are locked into.
The problem with this approach is that (esp. for hobby projects) updating stuff is a bit tedious. Let's say you have a relatively bare bones project in Rails or a PHP framework you have a couple of dependencies that people usually use together, so upgrading can be quick and painless.
I've now had it several times in the years-long lifespan of small clojure web projects that people have moved on and the thing (framework-ish) basically doesn't exist anymore and going by the issues it only had like 10 users in the first place.
It's not the end of the world, and fortunately there's not a lot of needless churn, but I guess I would prefer to have this "I am trusting project x and I only have to care about their releases (pre-testing all the moving parts) and then my 5 dependencies" and instead I have 20 dependencies/moving parts for my web app.
Yes, I'm lazy and I don't think it's a problem in an env where you have a proper dev workflow anyway.
Hot take: People on the internet make wild claims without any sources or data to back up what they're saying, and they think that's OK because they preface it with "hot take:" so whatever you reply, doesn't matter, because it was supposed to be a "hot take" that the author cannot really defend when needed.
If you need the next level of UX dynamism and or realtime updates and or multiplayer. I've handled a billion checkboxes[1] with clojure, sqlite and datastar (realtime hypermedia) just fine.
Following writing advice or post structuring guidelines is not in my job description at evalapply.org Luckily, Michael Hamburger offered a legitimate excuse in his classic (so I'm told) essay, "An Essay About Essays": https://substack.com/@bombaylitmag/p-162583447
Metabase is written in clojure, if you want to see the source code of a large web app
https://github.com/metabase/metabase
+1
NASA's Common Metadata Repository is worth exploring too https://github.com/nasa/Common-Metadata-Repository
It is a neat example of how an org can structure and manage multiple projects and services in a single git repository. They've use Leiningen to achieve their objective.
> The Common Metadata Repository (CMR) is an earth science metadata repository for NASA EOSDIS data. The CMR Search API provides access to this metadata.
> Building and Running the CMR
> The CMR is a system consisting of many services. The services can run individually or in a single process. Running in a single process makes local development easier because it avoids having to start many different processes. The sections below contain instructions for running the CMR as a single process or as many processes.
(edit: add relevant context for quick reference)
wow I had no idea Nasa used Clojure. I do remmeber them using quite a bit of Java so it's not terribly surprising
Thanks for the shout! I recommend this video about our Clojure journey: https://www.youtube.com/watch?v=vUe3slLHk20
We are also hiring Clojure devs: https://www.metabase.com/jobs
The ongoing discussion for Biff [1] prompted me to re-share my post because I'd like more people to understand this "other way". Outside Clojureville, it is not obvious most of these Clojure "frameworks" are not monoliths.
The consummate Clojurist's default (and very normal-feeling way) to build a web application (or any application for that matter) is to roll their own web stack from production-grade libraries.
Of course, this state of affairs is a double-edged sword, just like is true for traditional web frameworks. In my post, I try to go into the whys and the wherefores, building upward from first principles.
[1] Biff – a batteries-included web framework for Clojure https://news.ycombinator.com/item?id=44037426
A fantastic post! Enjoyed re-reading it.
I'd say the nice thing with the Clojure way of building your own stack is it becomes quite easy to swap parts out. On a previous project we swapped out our web server three different times with minimal changes (jetty -> aleph -> httpkit) as for the most part they all shared the same interface.
After a while you get good at seeing where you want things to be configurable and where you don't. It also gives you the confidence to roll your own micro stack/framework which means you are not dependent on third party aggregates to adopt new features.
Thank you for the kind appreciation. Made my day :)
Yes; next to the sheer stability of parts, their fungibility is a business-critical feature of the Clojure ecosystem. Of course said fungibility does not magically manifest. However the effort to get there is "not much", I'd say. The use of "system" libraries, with some well-reasoned module design brings it pretty close to magic.
As in the post, a fungible, production-grade part can be just a multimethod (e.g. the router in the post). Why? Because "production" comes in all sizes. A small SaaS with a few hundred customers may chug along happily with a bunch of functions.
This suggests to me that “production grade” isn’t much of a qualifier at all then.
You could just as well say that PHP has “production grade” functions.
Hm, I'm saying a given function can be production grade. I'm not saying all functions are production-grade. Also, I'm saying production comes in all sizes. If your micro-SaaS app gives you a livelihood, that's hella production --- real skin in game, real stakes in ground.
To analogise further...
- HackerNews is a "production" system, you would agree. Back in 2015, it was still true. Are flat files a "production grade" primary data storage choice? [1]
- Suppose your production service transacts a million requests an hour (say it is a short-link maker). Further, let's say it has only a handful of API endpoints. Do you really need a whole routing library for that, if a single multimethod does exactly what you need, correctly?
etc...
[1] https://news.ycombinator.com/item?id=9990630
(edit: add reference for flat file storage)
Right. And what I’m saying is that if “production grade” means all of these things, then it’s not a very meaningful qualifier.
My personal site is statically generated — it’s just a bunch of HTML files and some CSS. Do you not think it would be a bit pretentious of me to describe that as “production grade”?
If “production grade” simply means fit for purpose, then given the GP commenter’s reason for their initial web server swap, wouldn’t you say by definition that jetty is not production grade, since it doesn’t (didn’t?) support SSE?
I agree with you... "production grade" is in fact a meaningless term without saying what grade one's production needs to be.
A static site that serves is most definitely a production system. Perhaps one that could scale in traffic almost without end.
evalapply.org gets (to my continued amazement) 20K+ unique visitors a month when it's business as usual. On a busy HN day, it's easily that much in hours. I don't have to think about "scaling problems". I don't have one.
I'm being the realest real with you.
Your comment reminded me of this talk https://www.youtube.com/watch?v=MZy-SNswH2E.
The part about building just the functionality you need, using the bare minimum libraries etc stuck out.
> On a previous project we swapped out our web server three different times
Why is this desirable?
On all my projects over the past 10 years, I’ve swapped out the web server exactly zero times because the one I have works just fine. The parent comment describes these components as “production grade”, but then if that’s the case, what could be the reason for swapping them out other than self-indulgence?
It's not. But, the fact that we could and it was straight forward still amazes me.
In our case the first time we needed to, we needed SSE and at the time there was no ring-jetty async interface (it's a long time ago so I'm forgetting the details) so we moved to aleph. Much later, we wanted to try out http-kit (self-indulgence) as we were operating behind a proxy anyway for performance reasons and it made a significant difference.
If we'd just started with http-kit that would have been fine. I guess it comes down to what features you need.
Also I 100% agree it's something ideally you would want to avoid a in the case of databases for example so much performance is left on the table because for some reason we want to be able to swap between SQLite, postgres and mySQL. Which in practice you never want to do.
As the former maintainer of Aleph, I'm very surprised you ran into a situation where http-kit was faster. Or do you mean it was just preferable to develop for?
> Why is this desirable?
Generally, yes one would not want to swap out their web server willy-nilly...
Yet, this is one of those "YAGNI in 99% of your use cases", but when that 1% use case arises, a server swap would be far more desirable than a whole framework shift.
So while self-indulgence can certainly be a motive (and why not? as long as everyone's having a good time), may I offer a few more charitable reasons for this:
- programming API ergonomics
- performance
- application runtime model (servlets -> embedded server)
- security model
- application server features (websockets, comet?)
- binary size
- server configuration niceties
etc...
That said, a developer only has flexibility if it is built in from the get go.
A counterfactual would be to consider the set of developers who have had to put in ugly hacks because they can't just rip the web server out of the framework of choice they are locked into.
(edit: bullet list formatting)
> and why not?
Because I pay the people who work for me.
Well, if you're paying, you certainly get to set the rules. No self-indulgence, then. The other reasons I enumerated may still hold.
The problem with this approach is that (esp. for hobby projects) updating stuff is a bit tedious. Let's say you have a relatively bare bones project in Rails or a PHP framework you have a couple of dependencies that people usually use together, so upgrading can be quick and painless.
I've now had it several times in the years-long lifespan of small clojure web projects that people have moved on and the thing (framework-ish) basically doesn't exist anymore and going by the issues it only had like 10 users in the first place.
It's not the end of the world, and fortunately there's not a lot of needless churn, but I guess I would prefer to have this "I am trusting project x and I only have to care about their releases (pre-testing all the moving parts) and then my 5 dependencies" and instead I have 20 dependencies/moving parts for my web app.
Yes, I'm lazy and I don't think it's a problem in an env where you have a proper dev workflow anyway.
Hot take
Clojure seems to use 10x the memory and run at 1/5 the speed of something I would write in Go
Hot take: People on the internet make wild claims without any sources or data to back up what they're saying, and they think that's OK because they preface it with "hot take:" so whatever you reply, doesn't matter, because it was supposed to be a "hot take" that the author cannot really defend when needed.
Nope, please do a benchmark. It's almost saying Go is 5 times faster than Java.
a very lyrical post, i will reread at my leisure and try to apply the lessons to https://harcstack.org
that’s HTMX, Air, Red & Cro btw
that said … I am a true believer in HTMX for the right amount of UX dynamism and I don’t initially get solves that piece
If you need the next level of UX dynamism and or realtime updates and or multiplayer. I've handled a billion checkboxes[1] with clojure, sqlite and datastar (realtime hypermedia) just fine.
[1] https://checkboxes.andersmurphy.com
Thank you for the kind appreciation.
Following writing advice or post structuring guidelines is not in my job description at evalapply.org Luckily, Michael Hamburger offered a legitimate excuse in his classic (so I'm told) essay, "An Essay About Essays": https://substack.com/@bombaylitmag/p-162583447