Simon Willison's Weblog: webapps

What's the cheapest or free stack solution to deploy and experiment with a realtime application in 2016?

2016-08-16T09:02:00+00:00

My answer to What's the cheapest or free stack solution to deploy and experiment with a realtime application in 2016? on Quora

Heroku have a good free tier, and comprehensive support for deploying both Python and Node.js. If you are mainly interested in realtime I would suggest starting out with Node.js on Heroku. Depending on the complexity of your project you might even be able to use raw Node.js without adding something like Express.

Getting Started on Heroku with Node.js

Tags: hosting, programming, webapps, web-development, websockets, quora, heroku

How does one decide which Javascript framework (e.g. Node, Backbone, Angular) to use on any given project?

2014-03-11T17:08:00+00:00

My answer to How does one decide which Javascript framework (e.g. Node, Backbone, Angular) to use on any given project? on Quora

If you are just learning JavaScript, I suggest trying to work without any frameworks or libraries at all. Starting with something like Angular will make it much harder for you to learn the core language and browser APIs.

Once you've worked with raw JavaScript for a while, it's worth starting to learn jQuery - it's the basis of most of the popular JavaScript frameworks out there and understanding how it works will help reinforce a bunch of core JavaScript concepts.

If your priority is learning though, it's a good idea to start without jQuery. There are a surprising number of JavaScript developers out there who only really know how to program JavaScript using jQuery.

Tags: frameworks, javascript, nodejs, programming, webapps, web-development, quora, angular

What is the best payment provider for a web app with monthly subscription fees at its early stage?

2014-02-20T17:35:00+00:00

My answer to What is the best payment provider for a web app with monthly subscription fees at its early stage? on Quora

We found Stripe extremely easy to get started with for charging subscription payments.

Tags: startups, webapps, quora, stripe

Why did Twitter move away from being a single-page application?

2014-01-03T12:35:00+00:00

My answer to Why did Twitter move away from being a single-page application? on Quora

Twitter is still a single page application, it's just built properly now (one result of which is that you can't easily tell).

When Twitter first went single page, the only way to do so while still maintaining functioning hyperlinks and a working back button was to use a vile hack involving fragment URLs - things like https://twitter.com/#!simonw/sta...

This works, but comes with a whole bag of downsides:

Anyone who clicks a link to Twitter has to download several megabytes of JavaScript before the site can figure out what information it should be displaying. In the case of a tweet permalink that's several megabytes downloaded just to display a 140 character tweet!
Pages don't work at all without JavaScript.
A single JavaScript bug in an obscure mobile phone browser only available on Chinese handsets can prevent the site from loading any content at all.
Screen readers may fail to understand what's going on.
Only some search engines can index the pages, again using a nasty hack.
These aren't "real" hypertext URLs. If you perform a GET operation against them you don't get the content the URL refers to, you just get the Twitter homepage. This is bad for the health of the Web.
One relevant example: Quora's feature where a URL pasted in is automatically converted to the title of that page can't work if the URL doesn't point at real HTML, rather than a homepage and a big chunk of JavaScript.
These links start to spread. Other sites have to use them in links to Twitter, which means that even if Twitter gets rid of them they'll need to keep some JavaScript on their homepage that knows how to handle them forever.

These hashbang URLs are nasty, nasty hacks.

Thankfully the hashbang hack isn't necessary any more, thanks to a neat part of HTML5 called the History API. This allows JavaScript to update the visible URL in the browser's URL bar without actually navigating to a completely new page - essentially, it allows a single page application to pretend that it's a real website.

How does this fix the problems above? It means that Twitter can start serving real webpages again. Try running this command:

curl "https://twitter.com/simonw/status/419261896957505536"

It returns HTML! In fact, it returns the exact HTML needed to display a page that shows the content of that tweet. Then, at the very bottom of that HTML it includes a single script tag (loaded after the page has been displayed in the browser) which loads a huge bunch of JavaScript and converts that beautiful, light HTML page in to a giant single page web app.

Now, when you click on a link on the page, Twitter can fetch the new page contents using JavaScript and display it to you without doing a complete reload of the whole UI - but they can use the HTML5 history API to update the URL in the page so that, as far as you can tell, you've navigated somewhere else. It's a signle page web application hiding in plain sight.

But what about browsers that don't support HTML5 history, like older versions of IE? Here's the smart part: they don't get the fancy new JavaScript, they just get links which, when clicked, navigate them to a brand new page. This means that even browsers that don't support JavaScript at all can access and navigate Twitter, but browsers with JavaScript get a much enhanced experience.

Dan Webb, an engineer at Twitter, wrote some very insightful pieces about this issue. In It's About The Hashbangs (May 2011) he explained why he believed Twitter's hashbang implementation was the wrong way to go. A year later in May 2012 he published Improving performance on twitter.com on the official Twitter blog explaining the new history technique they were using and how it had dramatically improved the performance of the site.

In fairness to Twitter, I should point out that when they launched their hashbang single page app back in 2010 the HTML5 history API wasn't widely available - IE didn't support it at all, and I seem to remember Safari 2.0 had a bug which made their implementation unusable. Thankfully this is no longer the case today.

Tags: twitter, webapps, web-development, quora

Are Django versions released too often?

2013-12-17T09:52:00+00:00

My answer to Are Django versions released too often? on Quora

The Django release process is well documented (see Django’s release process) and has been specifically designed to address the concerns of developers who don't want to have to spend too much time keeping up to date with the latest version.

Minor releases happen roughly every nine months, and introduce new features usually without breaking backwards compatibility. If a feature is deprecated, it will raise a warning in the version that deprecates it, a louder warning in the next version (9 months later) and will be removed in the version after that (1.5 years after the feature removal was first announced).

If even that is too fast for you, you can stick with a "Long-term support" release, which get security fixes applied for 3+ years. Django 1.4 is the most recent of these, and will be supported until at least March 2015.

Read https://docs.djangoproject.com/e... for more information. Personally (and I've had no input at all in to the release process design) I think it's all extremely well thought out.

Tags: django, webapps, web-development, quora

What is currently the best technology stack for web scraping?

2013-12-10T17:32:00+00:00

My answer to What is currently the best technology stack for web scraping? on Quora

PhantomJS combined with CasperJS is pretty fantastic - it runs a full, headless copy of a Webkit browser so it can operate against a real DOM, execute JavaScript properly, even grab full rendered screenshots of areas of the page but is still easy to automate.

Tags: nodejs, webapps, quora

What are the best set of marketing/payment tools for use in startup with a subscription webapp?

2013-12-05T09:28:00+00:00

My answer to What are the best set of marketing/payment tools for use in startup with a subscription webapp? on Quora

Stripe has excellent support for subscription payments, including helping manage recurring payments and pro-rataing for upgrades to plans half way through a billing cycle etc. I believe BrainTree offers similar tools.

For email stuff, I strongly recommend looking in to Mailgun - they offer fantastic tools for advanced email sending. They don't have the same user-level engagement tools as something like Mixpanel or Intercom though.

Tags: startups, webapps, quora, stripe

What would be a good web application framework to use for running a small online retail business?

2013-11-23T11:50:00+00:00

My answer to What would be a good web application framework to use for running a small online retail business? on Quora

I'd look at hosted SaaS solutions rather than running your own. I haven't tried it myself but I've heard excellent things about www.shopify.com

Tags: startups, webapps, web-development, quora, ecommerce

PostgreSQL: How can I store images in a database? What existing products makes it easy for a user to upload photos into a general database?

2013-11-19T15:06:00+00:00

My answer to PostgreSQL: How can I store images in a database? What existing products makes it easy for a user to upload photos into a general database? on Quora

As a general rule, it's a bad idea to store images in a database. This is due to the large amount of space they take up, which can affect database read performance and will greatly increases the size of your backups, making them both take longer and cost more to store.

Instead, it's best to store the images elsewhere and then store a reference to them in your database. These days the easiest way of doing this is generally to use Amazon S3, which can cheaply and reliably store an unlimited number of images. Save them to S3, then store the S3 URL (or the bucket + key combination) in a string in your database row.

If you're determined to store them in a database you can do so using a BLOB field, or by base64 encoding them and storing them in a large text field (which will even further inflate the size of your tables).

Tags: databases, photos, postgresql, webapps, web-development, quora

Is there any way to generate thumbnails of web links like Facebook generates when we paste a link in status or comment?

2013-11-12T12:11:00+00:00

My answer to Is there any way to generate thumbnails of web links like Facebook generates when we paste a link in status or comment? on Quora

There are a bunch of services that can do this. I've successfully used http://url2png.com/ for this in the past.

Tags: facebook, webapps, quora

Is there a website or app that helps you track what you have completed each day?

2013-11-08T09:49:00+00:00

My answer to Is there a website or app that helps you track what you have completed each day? on Quora

I've tried a few solutions for this. Surprisingly the one that has stuck for me is Evernote - I keep a different document for each week (I tried a document per day but that was annoying to update, and meant I didn't look at my older notes as often) and each day I add a new header at the top of the document for that day. Being able to link through to other notes from my day summaries is useful too.

Tags: productivity, webapps, quora, motivation

Would you test your web app against simulated infrastructure failure?

2013-10-26T17:43:00+00:00

My answer to Would you test your web app against simulated infrastructure failure? on Quora

I think this is a pretty interesting idea - simulating these kind of conditions isn't easy so I imagine many teams don't bother. If it was good (really easy to get started with, great control and reporting tools, maybe helped set up the actual tests to replay) and I trusted the service I would definitely consider paying for it.

On the flip side, this isn't addressing a pain point that I actually have and feel - not simulating these kinds of failures doesn't keep me up at night, and my alternative would be not doing this at all. So take my "sounds interesting" with a big pinch of salt!

Tags: frameworks, programming, sysadmin, webapps, web-development, quora

What are some techniques, workflows, thought-processes etc. you would do as an experienced web-developer (front-end/back-end) that a new web developer might benefit from?

2013-09-16T10:18:00+00:00

My answer to What are some techniques, workflows, thought-processes etc. you would do as an experienced web-developer (front-end/back-end) that a new web developer might benefit from? on Quora

Learn the Chrome Developer Tools inside out, and live in them - especially the interactive JavaScript console, which can be a huge boon to your productivity.

Tags: programming, webapps, web-development, quora

What are key considerations when building behind the firewall web apps?

2013-09-15T15:24:00+00:00

My answer to What are key considerations when building behind the firewall web apps? on Quora

CSRF and XSS are still important: don't leave any security vulnerabilities which might allow an evil website out on the internet to run JavaScript that steals data from your behind-the-firewall web application.

Tags: csrf, enterprise, software-engineering, webapps, quora, saas

Should a beginner to web development start out with Node.Js?

2013-09-09T18:00:00+00:00

My answer to Should a beginner to web development start out with Node.Js? on Quora

Maybe. One of the things I like about Node.js is that the raw abstraction it provides over HTTP is much closer to how the actual protocol works than the abstractions provided many of the more widely used frameworks such as PHP, Django or Rails. That might actually make it an effective learning tool - I'd be interested in hearing from some web developers who learnt Node.js as their first server-side technology.

Tags: frameworks, javascript, nodejs, programming, webapps, web-development, quora

How can you run image processing in cloud code (Parse) ? If parse is not the best

2013-09-07T14:36:00+00:00

My answer to How can you run image processing in cloud code (Parse) ? If parse is not the best on Quora

There are dozens of web-based image resizing APIs that could help with this. Check out http://www.imgix.com/ for example.

Tags: javascript, webapps, web-development, quora

Why aren't there any popular web frameworks in C/C++ even though they are more performance intensive languages?

2013-08-05T14:17:00+00:00

My answer to Why aren't there any popular web frameworks in C/C++ even though they are more performance intensive languages? on Quora

Because it's very hard writing secure code in them due to the way they handle memory management, and web applications have an enormous number of potential paths for malicious input. It's much easier to write secure web application code in a higher level language (which isn't to say it's easy at all even then).

Tags: frameworks, webapps, web-development, quora

How do I choose between asynchronous web frameworks? My tech group is fairly language agnostic and we're trying to standardize on some technologies.

2013-07-31T10:15:00+00:00

My answer to How do I choose between asynchronous web frameworks? My tech group is fairly language agnostic and we're trying to standardize on some technologies. on Quora

Since they are all pretty close to each other and it sounds like your tech group's skills would support any of them, I would suggest having your tram build a simple prototype in all three so you can compare them for your own particular team and situation.

Tags: frameworks, javascript, nodejs, play, python, scala, tornado, webapps, quora

I have an idea for a website, with an idea for eventually having cashflow out of it? However, my skills are insufficient to build it and I have no money at all to hire someone - what's my best starting point?

2013-07-28T11:42:00+00:00

My answer to I have an idea for a website, with an idea for eventually having cashflow out of it? However, my skills are insufficient to build it and I have no money at all to hire someone - what's my best starting point? on Quora

You either need to learn some skills, or you need to earn some money. Sadly, having a good idea is not enough - you need to be able to execute it as well.

Tags: entrepreneurship, startups, webapps, web-development, quora

What technologies were used in developing web applications in the '90s?

2013-07-21T16:17:00+00:00

My answer to What technologies were used in developing web applications in the '90s? on Quora

Perl. Lots of Perl. There was a site called Matt's Script Archive which was full of terribly written Perl scripts for things like hit counters and form emailers and guest books. It was very popular.

Good lord... It still exists! http://www.scriptarchive.com/

Tags: internet, webapps, web-development, quora

What is the best web app for organize/group yours others web apps?

2013-07-21T11:34:00+00:00

My answer to What is the best web app for organize/group yours others web apps? on Quora

Have you tried using your browser's bookmarks?

Tags: webapps, quora

How is Web engineering related to Software engineering?

2013-07-15T09:13:00+00:00

My answer to How is Web engineering related to Software engineering? on Quora

I'd describe it as a subset of software engineering that deals with issues relating to the web - web application development, web protocols, browsers and so on.

Tags: computer-science, software-engineering, webapps, web-development, quora

What are useful tools for building out web app Infrastructure?

2013-06-27T18:52:00+00:00

My answer to What are useful tools for building out web app Infrastructure? on Quora

First, make sure you know your way around a command automation tool, such as Fabric (Python) or Capistrano (Ruby).

Then pick one of Puppet or Chef and learn it - neither are particularly well documented in my opinion, so pick the one that people you know use.

Tags: webapps, web-development, quora

What is the best way to deploy Django?

2013-06-16T10:03:00+00:00

My answer to What is the best way to deploy Django? on Quora

These days a popular and reliable method is to run gunicorn behind nginx. This tutorial includes notes on using upstart for process management which is handy if you are on Ubuntu: http://lincolnloop.com/django-be...

Tags: django, python, webapps, quora

What are some apps, problems you would suggest to solve a new python developer?

2012-11-07T17:17:00+00:00

My answer to What are some apps, problems you would suggest to solve a new python developer? on Quora

The best way to learn python in my opinion is using the interactive prompt. Install ipython (a massive improvement on the standard python shell) and use it to interactively solve some simple tasks - things like downloading a CSV file from the web using the urllib library, parsing it with the csv module, then poking around in the data using python list comprehensions and saving some of the results out to a JSON file.

Tags: django, python, webapps, web-development, quora

Is there a way to uniquely identify the device of the user that are connecting to a web app or a website?

2012-11-06T18:11:00+00:00

My answer to Is there a way to uniquely identify the device of the user that are connecting to a web app or a website? on Quora

If the device is a mobile phone running an app it's very easy to create a permanent identifier for the application/device - it can be generated once the first time the user runs the app, stored locally and then passed to Facebook with every future request. Think of it like a browser cookie, but unlike browsers the app developer can ensure the cookie is never deleted once it has been created.

Tags: webapps, quora

How should one go about designing a simple web based collaborative text editor?

2012-10-20T09:22:00+00:00

My answer to How should one go about designing a simple web based collaborative text editor? on Quora

Take a look at the open source ShareJS library, which implements a bunch of the fundamental (and extremely tricky) algorithms you need to get collaborative editing in the browser to work well: https://github.com/josephg/ShareJS/

Tags: programming, webapps, web-development, quora

How can I keep my website content exclusive?

2012-10-16T16:59:00+00:00

My answer to How can I keep my website content exclusive? on Quora

No, there isn't - people can always copy things if they really want to, it's part of the cost of doing business. If you were to publish a book people could always photocopy it.

If you're selling static content, I would suggest doing it as an ebook. People expect to spend money on ebooks, and there are ways of e.g. baking the user's email address in to a generated PDF file so that people feel more obliged not to share the ebook around (since it makes it obvious who the initial copy came from).

Tags: business, startups, webapps, web-development, quora

Why are web applications more popular than desktop applications with investors?

2012-10-13T17:19:00+00:00

My answer to Why are web applications more popular than desktop applications with investors? on Quora

How often do you install a new desktop application? How often do you try out a new web application?

Tags: funding, investing, webapps, quora, pitching

Did Mark Zuckerberg have any knowledge on building scalable social networks prior to starting work on Facebook?

2012-09-18T15:27:00+00:00

My answer to Did Mark Zuckerberg have any knowledge on building scalable social networks prior to starting work on Facebook? on Quora

I'm going to bet he didn't have this knowledge, simply because back when he launched Facebook in 2004 almost NO ONE had this knowledge - there simply weren't enough "web scale" products around for the patterns needed to run them to be widely discussed.

Today we're spoiled by the enormous amount of shared experience we have as a community in building large scalable systems, but back then it really was a case of each large scale site figuring everything out for themselves.

I bet he picked it up pretty fast though... nothing forces you to learn to scale faster than the experience of having your site stop working because too many people want to use it at the same time!

Tags: facebook, programming, scaling, social-networks, software-engineering, webapps, web-development, quora