Simian Enterprises

Find the closest in a list of numbers

Gary — Sat, 26 Apr 2014 00:27:54 +0000

For a while now I’ve been working on a core set of functionality that I include with every application. In fact, my applications extend a core app that contains a whole bunch of commonly used functionality – but that’s a whole other post.

I’d intended to blog about some of the components I’ve been writing as I go along, but as is so often the case I find myself moving onto new work before I’ve sat down to write about the last.

Today, I needed to pass a number to a function and return the closest number in a given list.
So for example, a list of ‘10,20,50,100,200,1000’ and a value of 42, should return 50.

It’s a simple requirement, but Google returned nothing & the mighty Ben Nadel hadn’t written a comprehensive post on how to do it along with speed tests and examples involving hot girls; so I wrote a function and added it to my ever increasing library.

As it’s small, I thought I’d share… in doing so, I realised I’ve not updated this site in way too long – so for this I apologise.

Here’s what I came up with – hopefully it’ll save someone a bit of time:

Lock wait timeout exceeded with ColdFusion & MySQL – Checked yer logs?

Gary — Sun, 31 Mar 2013 19:55:50 +0000

I just spent some time diagnosing an issue with inserting records into a MySQL database and I eventually tracked it down to a gotcha I think a lot of other people may run into.

The code in question does some fairly simple things – inside a it’ll insert a record into a table, read the record back and insert a record into a second table with contents from the first record.

Easy, right? Except every time I tried it, I’d get a timeout with the following error:
Lock wait timeout exceeded; try restarting transaction
Inserting the records manually worked fine, but not from CF.

After searching through the MySQL logs, I noticed that a second connection was made half way through the transaction – the remaining query would then time out and the transaction would roll back.
What I discovered is this:

is treated as a different connection to

Generally I won’t include usernames and passwords in my cfquery statements, but I’m connecting to a library that does by default. I’ve found that leaving the authentication fields blank will connect fine using those specified in the datasource, however it seems that MySQL considers it a different connection which will cause a problem in a transaction.
I’m running Railo 4 – the behaviour may differ on ACF.

So there you have it. A great argument for encapsulation and a nicely configured datasource object.

Opinion: The evolution of Affiliate Marketing

Gary — Mon, 05 Nov 2012 11:56:03 +0000

I often describe web development as an industry that’s inventing itself on a daily basis. One of the things I find fascinating is the evolution of an idea over time. When a system, model or process is invented online, it’s exposed to the entire world. The world inevitably has influence on it. The world changes it to fit new needs and desires often far beyond the scope of its original design.

There are many examples of this, and most are far more interesting than the one I’m going to talk about; but I got to thinking recently about the evolution of affiliate marketing.

The early days of Affiliate Marketing

I was first exposed to affiliate marketing somewhere around 2002. I was managing a small e-commerce startup and we were contacted by new kid on the block, Affiliate Future. The idea was simple enough – instead of paying for people to view our ad banners on various sites, we paid only for sales that were generated from those banners. It seemed too good to be true, but for once it wasn’t.

We signed up immediately and watched with glee as people began placing our ad banners all over their sites. Before too long, sales started rolling in. We’d have to pay 10% of the revenue back to our advertisers, but it was easy to take the view that these were new customers we’d probably never have gained without their ads.

It didn’t take too long for website owners to start taking advantage of the new advertising model. They’d put more effort into getting a user to click on an affiliate link by offering extra value. Perhaps they’d write a product review or offer a place for customer discussion, maybe a product or price comparison. By and large this was good for all involved. The ads only tended to work if they were placed on relevant sites, and we only had to pay for ads that worked. Users saw ads they were interested in, sites made the effort to target the ads to their users, and we got more sales as a result.

The next step: Affiliate sites

Then came the shopping directories. Huge sites filled with structured links to various e-commerce shops around the web. Some of them offered reviews and comparisons and whatever other extra value they could add to their offering. Shopping directories were nothing new, but the previous model of asking companies for a fee to be listed gave way to listing with affiliate links in order to get a percentage of the sales they generated. Generally the benefit of these directories for the companies listed on them was that they’d often do very well in the search engines, catching organic customers that the companies themselves may have missed.

Soon enough, shopping directories gave way to voucher sites. These sites would scour the internet looking for money off vouchers they could display to their users alongside an affiliate link. Now the companies would have to pay the percentage back to the advertiser, as well as the voucher they’d offered to the customer. This would obviously reduce their margins further; but ever eager to get new customers, companies began working with affiliates to offer exclusive vouchers for use on their sites – providing more value to the sites in question, whilst still keeping some control over what percentage is paid for a sale.

All of this was fine back in the days when companies had difficulty getting organic customers. When SEO was more about link triangulation than content, the shopping directories had the upper hand. Those days seem pretty much over. The search engines favour sites over directories, and customers are far more savvy than they were a decade ago.

The rise of cashback websites and a new kind of customer

Customers know how to compare prices across websites these days. Google product search makes that incredibly easy and since shop owners are able to add meta to their products to make use of unique identifiers such as barcodes, it’s easier for systems to trawl the internet and offer summaries of prices & delivery options etc.

So it seems inevitable that in the race to offer extra value, the only option left to the affiliate marketer is to pay for users to buy through them. So, the rise of the money back website.

The idea is that instead of the website owner taking the sale percentage for themselves, they offer it to the customer as an incentive to use their sites. It’s a great deal for the customer – they get money back for things they were going to buy anyway. The marketer in theory, gets more clicks through their links than they would have otherwise, as their offering is financially beneficial to the customer.

The only people that lose in this deal, is the e-commerce shops themselves. No longer are affiliate sales new customers that they’d not have gotten otherwise. Instead, they’re paying for many of their existing customers. Once a customer knows about a money back website, it makes sense for them to always buy through that site. If they’re going to spend £500 on a new tv, why not get paid £50 to do it?

A new way to shop is evolving. As customers have become used to comparing prices amongst various online retailers, so they are getting used to including money back websites in that comparison process.
The margins of e-commerce companies, often already razor thin; are being squeezed yet further by the need to literally pay for each sale. It’s no longer enough to just offer the lowest price or the best service – if a company doesn’t also offer cashback, they risk losing out to a competitor who does.
So it seems affiliate marketing no longer has a benefit to the companies using them. It’s less of a tool to generate new customers as it is a deal sweetener.

An example

As an example, I took a look at the TomTom Start 20 satnav.
Always my first port of call, I found this on Amazon at £88.95, free delivery and the always awesome Amazon customer service.
A quick check on Google product search and I can see that this is the cheapest price – however it’s shared by Currys, PC World and Halfords.
On to Quidco and I can see that Amazon don’t offer any cashback at all. Currys & PC World are offering 2%, Halfords 3%.
Halfords get my money, I get paid £2.67 for a purchase I was going to make anyway and Amazon have lost a sale by not having a competitive money back offer.

So who’s benefitting here?
Me, certainly. I’ve been paid money for nothing.
Quidco, absolutely. They’ve given back the entirety of their affiliate commission, however they charge £5pa for the use of the service. It’s less than another affiliate site might take, but I’m more likely to use Quidco than another site as they pay me more money to do so.
Halfords, perhaps. They’ve gained a sale that would ordinarily have gone to Amazon – however they had to pay for it. I’m not a new, loyal customer. Next time around I’ll do the same thing and my money will go to whomever gives me the best deal.

Who loses?
The small e-commerce company selling unique items. Affiliate marketing these days represents the purchase of sales, often from existing customers. If you’re a company selling unique items then there’s no scope for comparison shopping and the affiliate sales can cost more than they’re worth.

A final word

The Quidco link I posted above, is itself an affiliate link. If you click on it, sign up for an account and earn £5 in cashback, Quidco give me £2.50. Money for nothing. Everyone’s on the take, eh? Even me.

As a business owner, I recently cancelled all my affiliate programs. They no longer generate more than they cost.

Dynamically resize images with ColdThumbs

Gary — Mon, 12 Mar 2012 11:49:55 +0000

A few days ago I was coding up a WordPress theme for a client who intended to move away from WordPress.com.
One thing I noticed is that on the WordPress.com installation, images were dynamically resized by adding a parameter after the image file extension.
i.e. http://domain.com/images/an-image.jpg?w=400
Since many of the posts linked to images in this way, I had to find a way to replicate it into the WordPress installation on the local server. A quick Google search and I discovered ‘TimThumb‘, a php script for dynamically resizing images, very popular amongst WordPress theme developers.

Within a few minutes, I had the functionality I was looking for.
‘Very cool’, I thought to myself… So why can’t I do this in ColdFusion?

Enter: ColdThumbs!

Inspired by TimThumb, ColdThumbs is allows you to dynamically resize an image (local or external) to your desired dimensions.
The script will automatically create a cached version of your resized image and return the image as content.

I developed this as a bit of fun one Sunday (yeah, that’s how I roll) and have put it up as my first project on RIAForge. It’s working great for me, but I’d love to hear feedback or feature requests from anyone using it.

Check out the ColdThumbs page for full details.

Integrate WordPress into your ColdFusion app

Gary — Thu, 13 Jan 2011 12:57:04 +0000

Lately I’ve been working on a few ColdFusion apps that require a comprehensive blog system. While I could easily install Mango Blog or Blog CFC, nothing really rivals the functionality of WordPress when it comes to blog apps – but of course, WordPress is a PHP based system.

Installing WordPress alongside a ColdFusion app isn’t too difficult, but I need to customise the blog so that it looks identical to the rest of the site, and allow the user to switch seamlessly between the two.

This level of integration requires two main bits of functionality. Firstly I need to use ColdFusion to connect to the WordPress database and pull out articles and comments for use in summary blocks around the rest of the site, and secondly I need to be able to embed CFML templates directly into the WordPress blog to generate headers and footers whilst keeping any session based information such as login status, cart contents etc. – Essentially I need a CFINCLUDE equivalent for PHP.

Using ColdFusion to pull in WordPress content

I created a few components to pull data from a WordPress database. One to grab published articles, one to grab approved comments and one to remove any formatting and strip out extra HTML such as embedded images or videos.

The components are fairly self explanatory, and for the moment provide only basic functionality. It would be possible to build on these to provide greater integration, but for now these do the job nicely.

Pull published blog posts


    
	
    
	
	
	
		SELECT		wp_posts.*
		FROM		wp_posts

		WHERE		0 < 1
		
		
			AND		wp_posts.ID = 
		
        
        AND			wp_posts.post_type = "post"
        
        AND 		wp_posts.post_status = "publish"
        
        ORDER BY	wp_posts.post_date DESC
        
        
        	LIMIT #Arguments.iLimit#
		
        
		;
	

	
		// Return the query object
		return Local.qGetBlogPosts;

Pull approved comments


 
    
	
	
    

	
	
	
		SELECT		wp_comments.*,
        			wp_posts.post_name,
                    wp_posts.ID,
                    wp_posts.post_type,
                    wp_posts.post_status,
					wp_posts.post_title
                    
		FROM		wp_comments

		INNER JOIN	wp_posts
        ON			wp_comments.comment_post_ID = wp_posts.ID

		WHERE		0 < 1
		
		
			AND		wp_comments.comment_ID = 
		
        
		
			AND		wp_comments.comment_post_ID = 
		      

        AND			wp_comments.comment_approved = 1

        AND			wp_posts.post_type = "post"
        
        AND 		wp_posts.post_status = "publish"
        
        ORDER BY	wp_comments.comment_date DESC
        
        
        	LIMIT #Arguments.iLimit#
		
        
		;
	

	
		// Return the query object
		return Local.qGetBlogComments;

Strip HTML from posts

This function uses the ‘TagStripper’ custom tag by Rick Root / Ray Camden, available from http://www.cflib.org



	
	    

	

	
		// If we're only showing a summary, strip out everything after wordpress' '' tag.
		if (Arguments.bShowSummary) {
			Arguments.sString = REReplace(Arguments.sString, ".*", "");
		}
		
		// Stip out any HTML tags and return the output
		return tagStripper(Arguments.sString,'strip','p,strong,em,a');

Embedding CFM templates into WordPress / PHP

I don’t know a great deal about PHP, but I can’t find any equivalent to CFINCLUDE – even if I could it would be safe to assume that the PHP version wouldn’t be able to parse the CFML code contained in my CFM pages – what’s needed is some equivalent to CFHTTP that can hit the CFM page as a standard HTTP request, let the CFML server parse the code and return HTML that is then displayed within the PHP template.

Currently I’m using a function in PHP which wraps around ‘fsockopen’ to do exactly this. I found this function a while ago on another project – I’m not sure who wrote it.

With that function set up at the top of the php template, you can use the following code to embed a parsed CFM template.

echo fetchURL(http://pathto.your.cfm);

Now the only issue remaining is session data. As it’s the PHP server hitting your CFML template and not the user’s browser, the ColdFusion server is assigning a new session to PHP, and so any session specific data such as the contents of the user’s cart or login status, is lost in the displayed page.
So to combat this, we’ll need to check the user’s browser for a session cookie using PHP, and pass that along as a URL parameter to the CFML template.

Now we should find that the contents of the CFML page is parsed correctly, using the current CF session.

It’s worth noting that while this technique could be used to pull in your CFML header and footer template, providing a quick way to skin the blog without delving too deeply into the WordPress theme you’re using, this is not the ideal way to go. WordPress themes use variables in the head section of the document to deal with Meta information and other internal functionality, and these often provide hooks for WordPress plugins. If you replace this information with parsed HTML from your CFML templates, you may run into difficulties down the line.
A better method is to abstract individual elements of your site into separate CFML templates, and allow WordPress to create the finished page, pulling in elements where needed.

I’d love to hear what people think of these techniques and if anyone has a better way of integrating WordPress into their ColdFusion apps. Specifically, I’d be interested to know if anyone has combined the login functionality of WordPress with their CFML system – the holy grail, if you will!

Cache and display multiple Twitter feeds in ColdFusion – CFC

Gary — Fri, 19 Nov 2010 04:09:16 +0000

I’ve worked on a few sites recently that have required Twitter feeds to be pulled in and displayed on the homepage.
Thanks to the awesome tag this is a relatively easy task, but I’ve noticed a bit of an overhead when pulling the data in from Twitter’s servers.

I had a look around and came across this post by Simon Bingham, showing how to cache the results of a Twitter feed in the Application scope.

This worked well, but I needed to be able to pull in both outgoing tweets (using the Twitter ID) and incoming tweets (using a search string). This meant not only updating the CFC to take both types of argument and act accordingly, but also to allow mutiple feeds to be cached.
Building on Simon’s original, I’ve created a new CFC that does just that, and even has a bit of error handling for good measure.


	
	
	
	

	
		
		
		
			
						
		
	

		
	

		
		

			
			
			
			
			

				
				
					
				
					
				

					
					
						Application.stTwitterFeedCache[Arguments.sStructureName].data = QueryNew("Error");
						newRow = QueryAddRow(Application.stTwitterFeedCache[Arguments.sStructureName].data);
						QuerySetCell(Application.stTwitterFeedCache[Arguments.sStructureName].data, "Error", "Invalid query type entered. Please choose 1 for a Twitter ID, or 2 for a Search string.");
					

				

				
				

					
						Application.stTwitterFeedCache[Arguments.sStructureName].data = QueryNew("Error");
						newRow = QueryAddRow(Application.stTwitterFeedCache[Arguments.sStructureName].data);
						QuerySetCell(Application.stTwitterFeedCache[Arguments.sStructureName].data, "Error", "Unable to pull in feed. We all know ColdFusion is awesome, so it's probably Twitter's fault.");

Feasibly, this could be adjusted to cache any kind of feed – the trick is in the dynamic naming of the structures to allow for multiple feeds.

Hope someone out there finds it useful!

If ColdFusion is dead, it’s Adobe that killed it.

Gary — Wed, 15 Sep 2010 11:08:55 +0000

(Or, ‘Controvertial ColdFusion post #34124′)

I’ve been using ColdFusion happily for the last 6 years. I can honestly say that if it weren’t for the ease of learning ColdFusion, and the instant gratification to be gained from rapidly developing apps that interact with databases, I’d have never found my way into backend development.

Like any language however, ColdFusion is not without its problems. We can compare functionality until the proverbial cattle return to their domiciles, but what I’d like to talk about is the culture of elitism that surrounds ColdFusion.

‘Elitism?!’ I hear you cry, ‘But ColdFusion is so easy to pick up, and the community is so friendly and welcoming! That’s hardly elitist, is it?’.

Yes. You’re right… but what I’m concerned with is the difficulty ColdFusion developers face in the environment in which they work. Adobe have done a very good job of making ColdFusion inaccessible to the masses by focussing on enterprise clients, inadvertently turning ColdFusion into quite an exclusive club.
Don’t believe me? Reel off the names of some well known ColdFusion celebrities… Ray Camden, Ben Nadel, Ben Forta, Et al. I’ll bet if you’re a ColdFusion developer, you’ll know who those people are. You’ll have read their blog posts, be aware of projects they’ve done. You’ll probably be able to list 10 more without too much trouble…
Well, you shouldn’t be able to do that – because the list should be huge. If we were talking about PHP, there’d be maybe 5,000 people in that list. In ColdFusion there’s maybe 20.

Let me explain…
I’m a freelance web developer, who happens to favour ColdFusion as my back end language of choice. It should be as simple as deciding which language is right for the task, or which you feel most comfortable using, but with ColdFusion there’s more to consider.
First and foremost, I have to think about hosting the websites I create. If I were a PHP developer, I’d have the pick of pretty much any host you can think of. Hundreds of thousands of web hosting companies the world over and every single one of them offer PHP as standard with all their packages, right? For us ColdFusion developers though, it’s a much tougher choice. We have to find hosting companies that specifically cater to ColdFusion developers. Not an easy task. I’ve had accounts with ColdFusion hosting companies both here and in the US, and in my experience they’ve nearly all been so unreliable as to be detrimental to my business and reputation. It’s an issue we all face and I’ve had numerous conversations with other ColdFusion developers on the subject; finding a reliable ColdFusion hosting company is difficult. It’s a case of getting what you pay for, and a reliable CF host costs far more than a reliable PHP host, because a CF license is expensive for a hosting company to invest in.

CFML Developer = Part time sysadmin

The conclusion most of us come to eventually, is that it’s far easier to host our sites ourselves, getting a dedicated server or VPS with a reliable host, and installing ColdFusion onto it.
Now, aside from the huge cost of a ColdFusion licence, what this means for us is that nearly every ColdFusion developer has to have a sideline as a sysadmin. I’ve learned from experience that being a good sysadmin is a full time job on its own. This means there are a bunch of novice sysadmins out there, running their own servers through necessity – novice sysadmins mean insecure systems. Before you know it, we have ‘ColdFusion is insecure‘ rumours floating about the web. Just what we need.

What this also means is less choice for our clients. With every new client, I have to have the same conversation: “I need to let you know that if you take up my services, we will need to move your site to another hosting company, or host the site on my own servers“. This isn’t so much of a problem for enterprise clients, but at my end of the scale, speaking to small companies with even smaller budgets and a knowledge of how the web works that is even smaller still, these words are strange and daunting and often a dealbreaker.

Most of my clients have never heard of ColdFusion. They’ve never heard of PHP either, but wheras with a PHP developer they could continue to live in blissful ignorance, with a ColdFusion based solution they will quickly learn that ColdFusion requires a hefty licence fee or at least more expensive hosting.

The situation gets worse too, because apparently ColdFusion cannot reliably be run in a shared hosting environment. “What?? That’s ridiculous!!!” Yes it is, and I would never have thought it were it not for a conversation I was having recently with Andy Allan of Fuzzy Orange. (We all know who he is too, right? See? CF… good if you want to be famous, bad if you want an easy life.) All this time I was thinking that most CF hosting companies were complete morons, incapable of offering a reliable service, but in fact it seems that they’re at a disadvantage right from the beginning because CF Enterprise sandboxing has numerous problems that plague the everyday plight of the sysadmin.

Now, it’s my contention that if this is the case, it should be a huge priority for Adobe, but apparently it’s not.

The big picture

ColdFusion is less used than PHP and ASP. That’s a fact. I’ve heard a million arguments about enterprise users and fortune 500 companies, etc. We as a community are constantly battling against the ‘Coldfusion is Dead’ bullshit and frankly, I’m with you guys. It’s very much alive, very much out there and being used all over the place… great, but you can’t deny that PHP is the most widely used back end language.

This has a knock-on effect for us. It makes it more difficult to sell ColdFusion as a solution to clients. It’s easy for our competition to mis-inform. I’ve lost clients to PHP because of higher hosting costs, I’ve even lost a client to an SEO cowboy who claimed that ColdFusion was ‘Bad for SEO’. The fact is that ColdFusion is less known and less used than the free alternatives, and we as a community are constantly trying to raise awareness, spread the word and in many cases, defend our beloved language.

How exactly can we make ColdFusion more prevalent when so few hosting companies offer it as an option? How can we encourage new developers towards CFML, when they’ll also need to learn about how to set up and manage servers in order to use it reliably?
The answer I usually hear is that if there’s enough demand for ColdFusion, then bigger hosting companies will offer it as a service. But that’s a chicken / egg argument. How will the demand grow, unless it’s there as an option for the masses? How many PHP developers began their learning solely because they needed to update or configure a PHP app they were using such as WordPress or PHPBB. Why are there no real CF ‘killer apps’ like WordPress and PHPBB?

The way I see it, CF will remain mostly at enterprise level, behind the scenes, until such time as it comes as standard with many, many more hosting packages. And this won’t happen as long as it’ll cause a support headache for the big hosting companies. We can all argue about the price of CF, but that’s not really the concern for the big hosts, they can get bulk discounts anyway… their problem is that if they offer CF in a shared hosting environment, they will have reliability problems, and will eventually have someone like me announcing to the world that they’re a bunch of useless wankers.

Railo to the rescue!

Sure, I know there are a lot of people out there who will tell me that Railo is the answer to all of this. It’s free, it’s fast, it’s open source and the community are great. I agree on all counts, and indeed I’m now offering reliable hosting to my clients with my own servers running Railo. Brilliant.
But it’s not the answer to everything.

Firstly, Railo has only really come into its own, at least in my opinion, over the last year. About 18 months ago I tried to make the move to Railo and there were just too many bugs and inconsistencies for it to be a viable solution. It simply wasn’t capable of running my apps, or at least not without some serious reworking of a lot of functionality. V3.1.2.001 has changed all that, and it’s now able to handle damn near everything I throw at it – but it still contains bugs. The Railo community are great and bugs, once identified, are fixed quickly – but this means that I as a developer have to spend some of my time testing & reporting. Sometimes I run into a brick wall with something I’m working on, and have to wait for a fix from the Railo team before I can move forward with it. That’s fine, it’s the nature of open source and I’m very happy to be contributing to the project, finding issues in a live real-world environment – but it’s still unpaid man-hours that I wouldn’t have to deal with, were I using Adobe ColdFusion.

Secondly, while there are community members out there doing great work on installer projects, posting tutorials and the like, there’s still a much steeper learning curve in the setup of Railo compared to ColdFusion. What you’re rewarded with is a much faster CFML engine, Railo’s lighting fast compared to ACF, but the point I’ve been making throughout this post rant is that I don’t want to be a sysadmin, I don’t want to have to learn about application servers and web servers, I just want to code bloody CFML!

What’s the answer?

Hell, I don’t know… but it occurs to me that the CF community is fighting a constant war against misinformation, defending CFML to clients and colleagues alike. The community as a whole seems to want to raise awareness of ColdFusion, and if indeed this is the case, then I think Adobe needs to take a serious look at who they’re aiming at.
As Andy pointed out, most of Adobe’s CF customers are enterprise, so they may not see hosting companies as a big sales generator… but it never will be be unless the sandboxing issues are resolved. Meanwhile, it’s people like us who suffer. This isn’t a concern if you’re a CF developer working in a bluechip traded company with a huge IT budget, but if you’re a freelance developer with a love of ColdFusion, this should matter to you and I think it’s time Adobe looked at the big picture – A lack of reliable CF hosts is a barrier to entry, which means less CF developers in the wild, and a harder time for those already out there.

Tutorial: Setting up a production Windows 2008 server with IIS7 & Railo

Gary — Tue, 29 Jun 2010 11:14:39 +0000

Prologue:

In which Gary explains his new found appreciation for anyone who calls themselves a sysadmin.

I’m not a sysadmin. I like to make websites. It’s what I do, what I’ve always done.
Dealing with servers is the un-planned love child of my long term affair with website development. A horrid child that demands constant attention and gives nothing back in return.

In the past, I would point clients in the direction of a decent web host and let them get on with it, but as it turns out these clients would still phone me as the first point of contact when their servers went down, making me a mediator between them and their hosts. Frankly, I figured if I’m spending my time doing this anyway, I may as well get paid into the bargain.

Well, after four years of hosting client’s websites I can quite categorically state that sysadmins have one of the most difficult jobs imaginable. Anything can go wrong, at any time. Running a tight system involves research, dedication, and genuine enjoyment of high level tinkering.

If I’m ever in a position to employ a sysadmin, they will be treated well. I will make them tea. And cake. And give them sympathy.
Recently, after a long and gruelling battle with the most unreliable hosting company I’ve ever used, I finally took the plunge and set up my own Windows VPS using IIS7 & Railo.

The decision making process:

In which Gary explains and attempts to defend, his reasoning.

“Why use IIS?!” I hear you scream at your monitors, which proves that I do in fact possess super-human hearing abilities… Well, as I’ve taken great pains to explain, I am not a sysadmin. Years ago I ran a linux server that was hacked to pieces because I didn’t know enough about securing said system.

Windows and IIS, for all its faults – and there are a great many – is a GUI based system. What this means in the real world is that there’s an icon for everything. So, as a non sysadmin trying to setup and run a server, having an icon for every conceivable thing I’d want to do, serves to actually indicate what I might want to do in a way that a collection of config files simply doesn’t.

Also, having used it for a while now, I have to say that IIS7 isn’t half bad. It’s a damn sight better than IIS5 and IIS6, both of which I’ve had the dubious pleasure of using during the course of my career. Also, Server 2008 is a lot nicer than previous incarnations. Sitting here typing this post in Word 2007, on Windows 7, talking about Server 2008, I have to say that Microsoft seem to be taking an interest in user experience. They’re a long way off Apple, but they’re finally doing something half decent. Well done them.

“So why Railo?”
Well, initially this was an issue of cost. Frankly I can’t afford to shell out for an enterprise CF licence. I’m not a big company, I’m one man making awesome websites for very small companies. So I thought I’d give open source a go, and see how it compares.

I have to say I tried Railo about a year ago and found that it was lacking in too many places to be a viable solution. There were a few tags missing, a few things it just seemed to handle differently, and given that my sites generally like to nudge a few boundaries, it just wasn’t up to the task.

Well, all that has changed. Railo this time around was an absolute pleasure. A steeper learning curve to set up compared to Adobe ColdFusion, granted, but what you’re rewarded with is a blisteringly fast ColdFusion experience, a much higher level of control over your environment, entry into a knowledgeable and welcoming community and a general feeling of wellbeing that can only be gained from not giving Adobe three grand of your hard earned cash.

Hacking my server 101:

In which Gary, in an attempt to offer help to others in a similar situation, provides a step by step guide to the setup of his production server, and hopes that malicious people don’t use it to bring the thing crashing to its knees.

In the beginning, there was the welcome screen and the server was without services.

So, let’s assume that you’ve got yourself a nice sparkling new Windows 2008 server/vps set up.
Depending on how your hosting company has set this up, you may need to install various Windows updates, so it’s best you do this before anything else. Go to Windows update. Go directly to Windows update. Do not pass go, do not collect $200.
Usually I’d advocate installing Anti-Virus software next, but bitter experience tells me it’s best to install that at the end, after installing the various components we’ll need to get the server working.

And the sysadmin said, “Let there be services, and let the server use them to serve pages to the masses”.

Roles:

Firstly you’ll need to install IIS and various roles. I’m not any kind of an expert on this, so I won’t suggest which roles are right for you. Best practice as I understand it, is to only install what you need for the task at hand, thus minimising the chance of attack.

I’m using Windows 2008 Web Edition, which comes with literally nothing but IIS. If you’ve got a better version of 2008, you may want to install the DNS role, email, any number of others. Go wild, have fun.

One way or another you’ll need to install all the IIS6 roles, as well as the IIS management role.

Email:

I’m using MailEnable, basically because it’s free and does the job. When I spoke to my new hosting company about this, they strongly suggested trying ‘SmarterMail’ as a better alternative. Give them a look and make a decision.
Guides to install MailEnable are here:
http://www.mailenable.com/support/MailEnable_Installation_Guide.pdf
http://www.mailenable.com/support/MailEnable_Quick_Start_Guide.pdf

You’ll need to open ports on your firewall to enable… umm… MailEnable… so don’t forget. This includes Windows Firewall as well as any hardware firewall you may have set up.
Incoming: 110 & 25
Outgoing: 25

DNS:

If you want to use your server as a nameserver, (and frankly, if you need to read this tutorial to set up a production server, you’re unlikely to be the kind of person who’ll have a separate DNS server, so I’m talking to you!) you’ll need some kind of DNS service.

I’m using the cut down cheap-ass ‘Web Server Edition’ of 2008 server for which Microsoft have deemed fit to not include DNS services. So, like most of the planet, I’m using Bind.
It’s fiddly to get your head around if you’re unfamiliar with the concepts of DNS, but once you get the hang of what’s happening it’s all very straight forward.
Take a look at this tutorial: http://alex.charrett.com/bind-on-windows-mainmenu-3

You’ll also need to register your domain with the nameserver authority – or more accurately you’ll need to get your domain registrar to do this. A lot of registrars have an automated section in their control panels where you can do this. Others you’ll have to email. But basically, you need to have a domain name pointing to your IP address on the main database of nameservers. This usually takes about 24 hours.

You’ll also need to open your firewall up for BIND services:
Port 53, inbound and outbound, both TCP and UDP

FTP:

Initially, I thought it best to use the IIS built in FTP 7.5… However, after much messing about I decided it wasn’t up to the task.
If, like me, you like to have a ‘private’ folder outside the webroot to keep cfcs and the like, I’d recommend ditching it for FileZilla server.
I’ve left instructions for FTP 7.5 here for posterity.

FTP 7.5
Windows server 2008 has a new FTP module, FTP 7.5. It’s supposedly better for a million reasons, but what I like about it is you no longer have to create windows users to authenticate an FTP session. I’ve never much liked windows user permissions, gimme a username and a password and I’m happy.

It does take a bit of messing about to get the new user system working though. A very handy tutorial exists here:
http://learn.iis.net/page.aspx/321/configure-ftp-with-iis-7-manager-authentication

FileZilla
The FileZilla server is a much nicer solution in my opinion, it works much as I’d expect an FTP server to work, and doesn’t require as much messing about with IIS users and the like.
There’s not much in the way of tutorials out there, but it’s so simple to set up I’d be surprised if you need one. Nevertheless, here’s a link to one for good measure:
http://www.raymond.cc/blog/archives/2007/10/19/how-to-setup-ftp-server-on-windows/

Once you’ve set that up, you’ll want to set up the firewall for FTP.
That means opening up port 21, as well as enabling PASV mode with the following command:
netsh advfirewall set global StatefulFtp enable.

PHP:

Yeah, even though we’re coding awesomeness in CFML, there’s always going to be some client who wants to use PHP for something or other. Usually it’ll be a WordPress installation, which, I’m sorry, is just a better blogging platform than the CF offerings. There, I said it. I feel better.

This tutorial should guide you through the pain:
http://www.trainsignaltraining.com/iis-7-install-fastcgi-php/2008-09-04

IIS modules:

Personally, I found that I needed the IIS7 Administration Pack. I can’t for the life of me remember why, but I’d suggest you just install it and stop asking questions. ‘k? ‘k.
http://www.iis.net/download/AdministrationPack

The URL Rewrite module is a bit more obvious – you want this. It allows us IIS users to do what Apache bods have been doing happily – and somewhat smugly I’ll add – for years… Rewrite URLs using RegEx. Again, if you’re slapping WordPress on any of your domains, you’ll need this for friendly URLs.
http://learn.iis.net/page.aspx/460/using-url-rewrite-module
http://www.iis.net/download/URLRewrite

Perl:

Well sure, no-one uses Perl anymore, but it comes in handy having it on your server. Especially if you intend to install a stats package like AWStats. As it happens, AWStats is such a bitch to get working correctly that I wouldn’t bother, but still… Perl = good.
http://www.activestate.com/activeperl/downloads/
There are a few different things you may have to do to get this running on IIS7, including enabling a 32-bit application pool if your server is 64bit. Check out the instructions here:
http://blogs.iis.net/wadeh/archive/2009/04/13/running-perl-on-iis-7.aspx

AWStats:

Initially I had detailed instructions on installing AWStats here, but basically… just don’t bother. Get all your clients’ sites on Google Analytics. It’s a better package anyway. Honestly, you’ll thank me for that advice.

MySQL:

MySQL is fairly straightforward to install, but if you need a hand explaining the various options, there’s a tutorial here:
http://www.trainsignaltraining.com/install-mysql-on-iis7/2008-09-10/

phpMyAdmin:

If you need it, now would be the time to install phpMyAdmin.
http://www.trainsignaltraining.com/install-phpmyadmin-on-iis7-and-server-2008/2008-09-16/

One thing this tutorial isn’t clear on, is setting up the linked-tables feature. Several comments note the error, but none show how to fix it. You need to create a database specifically for these features. Instructions here:
http://www.phpmyadmin.net/documentation/#linked-tables

And the sysadmin looked at the server, and saw that it was good. And the sysadmin said “Let CFML pages be served, that web developers may rapidly develop and deploy applications”.

Installing Railo on Tomcat on IIS7 with multiple sites… I could write out step by step instructions, but why re-invent the wheel? I followed an excellent tutorial by Doug Boude, and you should too:
http://www.dougboude.com/blog/1/2009/09/Railo-31-on-Windows-Server-2008-and-IIS7–Part-2-of-3.cfm

At some point, this guide will ask you to download a DLL file to connect Tomcat to Railo – the URL in the guide is out of date, but I found the DLL here: http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/

It’s also worth noting that the guide expects you to be using version 1.2.28 of the ISAPI Redirect URL, however there is a newer version, 1.2.30 available. Do NOT use this version! It took me a while to figure out, but 1.2.30 makes everything run incredibly slowly. Don’t ask me why! I moved back down to 1.2.28 and everything worked fine.

One thing that this guide doesn’t mention, is how to handle default documents. You can see in the comments, a suggestion that adding ‘/*=wlb’ to the worker properties file will push all files through to Tomcat, which will indeed handle default documents. However, this will also put all static files, images, js and the like, through to Tomcat. Not only is this overkill (although I didn’t notice a performance hit), but also in my experience Tomcat has difficulty returning static files 100% of the time. I noticed certain images and JS returned as 404 errors, even though they existed.
Cue a helpful bit of info supplied by the Google Railo group.
http://groups.google.com/group/railo/browse_thread/thread/8706a5a4b025f393
This will allow you to set up default documents the *correct* way, leaving your static files to be handled by IIS as they should be. Don’t skip this step, it’s important!

And the server was able to serve CFML and the sysadmin saw that it was good. And the sysadmin said “Let there be protection, that viri may not penetrate the goodness of the server”

Finally, you can install Anti Virus. I’m using McAfee, since I happen to have a copy I’m not using.
Be careful to edit your preferences to ensure that not all files are being scanned – otherwise it’ll kill the server very quickly. Choose to scan files on writing to disk, and only the ‘default files plus additional’ option.
You’ll also want to stop McAfee from blocking any emails sent by the server. In the access protection settings, under the rule for blocking mass email worms, add ‘Tomcat6.exe’ to the list of excluded services.

And the sysadmin saw that McAfee was using way too many system resources on boot, but that it settled down after a few minutes, and the sysadmin saw that it was good enough. And the sysadmin said “Let’s make sure I don’t have to do this shit again!”.

A backup system is a good idea. If you’re smart, you’re using online version control and so losing the files and files of code you’ve carefully crafted, simply isn’t an issue. If not, check out Kiln or Git. I use Kiln and it’s an absolute pleasure. Don’t use SVN, because it’ll cause no end of hassle in the long run… and who needs a VCS subfolder in every single folder in their app anyway?

For MySQL, I’m using a batch script that exports every database, once a day. I built on top of the script in this guide: http://www.iis-aid.com/articles/how_to_guides/backing_mysql_automatically_using_batch_file
My version only holds one backup of each database on the server per day, but FTPs them down to my dev server which holds multiple copies. Have a look here: MySQL Backup Script

And the sysadmin looked at the server and saw that it was good. That it served CFML pages extrodinarily fast, and the sysadmin was pleased. The sysadmin rested. And was rewarded with cake.

Mercurial tool to export changed files

Gary — Tue, 27 Apr 2010 15:58:08 +0000

Recently, my friend Chris Brown convinced me to move my source control from SVN to Mercurial (or more specifically, Kiln).
I found Mercurial to be infinitely better than SVN in almost all resepects, but there was one thing missing that I could do with Tortoise SVN but not Tortoise HG, and that was to export any changed files from one revision to the next – something which as a web developer, I have to do very often.

Well, Chris came to the rescue and created a simple program to do exactly that. So here’s Chris to explain the program and how it works:
Download links are at the bottom of the page.

Hi,

My name is Chris Brown and I am a big fan of FogBugz so when FogCreek announced Kiln (a mercurial client) I immediately jumped on the bandwagon moved to Kiln and have never looked back. The Kiln Mercurial solution solved all the issues I had with merging and speed in Subversion and I was a very happy man indeed.

Then one day whilst chatting to my good friend Gary @ Simian Enterprises (who has kindly let me blog via his site on this occasion) I mentioned Kiln to him and proceeded to wax lyrical about how great it was until his ears were bleeding and to my surprise I had convinced him to give Kiln a try.

Now Gary was at first very happy and was as thrilled as me at the beautiful Kiln web experience and TortoiseHg integration, however there was one nagging feature he used a lot in Subversion which does not exist within the Mercurial system, and after doing a bit of googling it seems a few other people hit the same snag.

It is not possible with TortoiseHg to export a copy of all files changed since a specified revision. Now to most this would not cause much of an issue, I myself have never had this need, and do not think I will, but for those working on big complex websites like Gary does this is a big issue.

When uploading the new version of a site (usually by FTP) this can take some time, and I am sure most people will agree that uploading files with no changes is nothing but a waste of time. Being able to export just the files changed since the last update and only updating these makes a lot more sense and will result in far fewer files being transferred and as mentioned TortoiseSVN has this capability as standard.

With any luck one day the TortoiseHg system can also have this command, until then I have written a small application in Visual Studio 2010 which does what Gary needed and hopefully it will be of use to other Kiln/Mercurial users too.

Please note: this is one of those programs I knocked up very quickly, it does not have much error handling and the code is not that tidy, however it does work and I have provided both an installer for the app and the original Source Code, please feel free to amend as needed or build on this to meet your needs.

Below are details of each input and what it is used for and what the buttons do, this currently checks for changes between a revision and tip and assumes the current working folder is updated to Tip.

Root Directory of mercurial repository: should point to the path of the root of your repo (where .hg lives)
From Revision #: the revision number to check from
Export target folder: the folder to copy changed files to (must be blank)
List Changed Files: lists the files in the list box that will be copied based on current settings.
Export Changed Files: same as above but also copied to the target folder/
Close: does what it says on the tin.

If you like this little utiity and would like to make any comments you can find me on Twitter (@CB46)
Thanks
Chris Brown

Download: Windows Installer | Source

Combatting misinformation in web design

Gary — Thu, 11 Mar 2010 13:40:30 +0000

I received an email from a client recently, informing me that they have hired someone to redevelop their entire website in php, as they have been informed by their SEO company that ColdFusion is ‘bad for search engines’.

Frankly, I think it’s astounding that any SEO company could make such an assertion, anyone in the industry would immediately understand just how ridiculous this statement is – but unfortunaty our clients are not experts and can only make their decisions based on the advice they receive from the people who claim to be. My clients in this case have made an informed choice, based on patently false information…

So I’d like to state definitively: ColdFusion has nothing whatsoever to do with SEO… Neither does php, asp, ruby, python, perl, or in fact any back end language at all…

Search engines read the ‘mark-up’ of a website, that is to say the HTML that anyone can see by clicking ‘view source’ in the browser… A back end language such as ColdFusion, php or asp, will generate HTML mark-up according to the templates that have been coded by a developer. It is entirely possible for any of these back end languages to generate identical mark-up.
My clients in this case have paid to have their website recoded, but the HTML produced will be exactly the same as their existing website and so their search engine results will be completely unaffected… In fact, since the file extensions will change on every page, many incoming links will no longer work, so unless their new developer puts 301 redirects in place, they will most likely drop in the rankings.

As a developer working primarily with ColdFusion, it’s easy to feel angry that my clients have been mislead into believing that the work I have done for them is somehow inferior because of the language used, and indeed if I knew the name of the SEO company involved I would be in contact with them directly to argue the issue as well as naming them here… But what’s worrying is that it’s really the clients that are suffering. Through ignorance of how the Internet works, they have been led down a route that is both costly and futile, by a company that either has no knowledge of their own industry, or even more worrying, is ruthless enough to take advantage of the ignorance of their clients.

I only want what’s best for the people I work for. I want their sites to work well, to become popular, to generate revenue – and I try to give the best advice I can to help clients understand what can be
gained from their web presence. I’m sure we all do… But in a technical industry such as ours, one that combines so many different disciplines, one that every virtually every business needs to interact
with and yet very few understand, how can we combat misinformation like this?

The average client doesn’t need or want to understand how the Internet works. Mention CSS, JavaScript, back end software, web standards, etc. to the average client and they will at best, stare at
you blankly… At worst pretend they know what you’re talking about when in fact they haven’t a clue. In my experience, the best clients to work with are those that have enough knowledge of the web to understand that user experience is key, that copy is important, and that their website is an ongoing project. They don’t need to know the intricacies of code, servers and the like, but they need to trust
us to make certain decisions on their behalf.

Perhaps it’s time we tried to educate our clients. I don’t know how much information is out there to explain the basics. Maybe we should have a simple guide explaining how websites are put together that we can give to clients at the beginning of new projects… Either that or perhaps SEO companies should be licensed and regulated!!!

As I write, I’m not entirely sure what the point of this post is, but I feel it’s an issue not generally discussed and I’d be interested to know what others think. Perhaps the larger agencies don’t run into
this problem due to their reputation as experts or their tendency to work with bigger clients; but at my level, developing for small companies with little or no online strategy, half the battle is guiding them in the right direction.

So how do we compete with liars???