Word and Phrase Repetition — the Proximity Question

I can write an algorithm to do pretty much anything with a simple list of words–which, at its most basic level, is what a short story, novel, screenplay, blog post, article or essay is.

Count the number of times a word appears–no problem. Count the number of times each phrase appears–a little trickier, but still no problem. And I can see the use of these checks to writers. Are you relying a little too much on a particular word or phrase? Do you have a habit of beginning one sentence too many with a particular word or phrase? Has an unusual or rarely used word turned up four times in your 80,000 word novel?

This is what computers do. It’s what they’re for. Spotting patterns, counting things, making it possible to draw conclusions by presenting facts that are difficult for the human eye to spot.

But these checks are only as useful as they are useful. Adding complexity just because you can–not a good idea.

I’ve been asked a few times by SmartEdit users to improve the word and phrase frequency counters to show the proximity of repetitions. Other software does this, so why doesn’t SmartEdit?

It’s been suggested that using the phrase “On the other hand” 250 times in a novel might not, in itself, be a problem, not if those 250 occurrences are spread out evenly over the entire work. Not if one use of the phrase doesn’t pop up within a page of another.

I disagree. Using that particular phrase half a dozen times in a full length novel, not to mention 250 times, is a problem regardless of how close one occurrence is to another.

The proximity checker?

Unless there’s real value for the writer, there’s no gain in adding such complexity. Data for the sake of data, complexity without purpose. Rather than add value, it detracts–from the software and from the results. It makes the writer’s job of editing more difficult, not less; makes them question deliberate use of phrase and word repetition.

The power of this check is not in catching excessive word or phrase usage in a paragraph or on a single printed page, it’s in catching the minor repetitions of words and phrases that jar the reader when encountered more than once.

I remember reading a novel where the writer used the word opined once in every chapter. Fair enough, they were long chapters, and the writer may have felt comfortable with the distance between each opined. But for me, the reader, every time I encountered that dreadful dialog tag was a slap in the face. What was he thinking? Had I not been inflicted with enough opines to last a dozen life times? Can his characters not simply say something?

Why write about this today?

I’m reading Jeanette Winterson’s children’s novel Tanglewreck, and have just reached a passage describing a character who I’m guessing will prove to be one of the bad guys. This wonderful description of Abel Darkwater would be flagged in bright red flashing lights by any sort of word repetition proximity checker.

Abel Darkwater was a round man.

He had a round face, and a round body, and round rings on his round fingers. The gold loops of his pocket-watch chain were round, and when he drew out his watch, which he did as the taxi pulled up to the door, his watch was round and fat and gold.

But the repetition is deliberate–obviously. As is the phrase repetition in Dr. King’s I Have a Dream speech.

Repetition, when those words or phrases are in near proximity to each other, is not the purpose of the various repetition checks in SmartEdit. Their purpose is to catch those dreadful opines, scattered like acne over the length of a novel, forever popping at the wrong times and spewing all sorts of unpleasantness upon the reader.

So, for now, the word and phrase repetition counters in SmartEdit will remain as they are. Because even though the algorithm could be written without much difficulty, I’m not convinced it should be.

07 Aug 2013

Software’s Place in the Editing Process

One of the common criticisms levied against self-published books in recent months has been the lack of professional editing that even a cursory glance at a few Kindle published novels reveals. While this criticism is not always warranted, in many cases it is.

The level of editing required after completion of a first draft varies from writer to writer. Some writers are able to deliver two or three polished novels a year, while less productive writers take years to finish a single book–grinding through years of continuous editing until they are happy with the finished product. Every five years or so Kazuo Ishiguro produces a new work. It’s invariably short-listed for one literary prize or another, and it’s always well worth the wait.

Why do so many self-published authors fail when it comes to properly editing their work? Don’t they want their novel to be as good as it could be?

I see three reasons.

Firstly, the new self-publishing industry has impacted the writing and publishing worlds like a live grenade tossed through the window of an antiques shop. Stories abound of writers publishing their own material as soon as it’s finished, selling thousands of copies a month, and leveraging those sales to land traditional publishing contracts and movie deals.

Eager young writers want to emulate this success, and they want to do so today. Why wait a few years until they become more accomplished when they can learn on the job? This approach is championed by gurus such as Dean Wesley Smith and others, who advocate publishing fast and furiously, all the time training yourself to write better and better. Not surprisingly, editing is often a casualty of this rapid fire approach to self-publishing.

Secondly, many writers who have never been published by traditional, dead-tree publishing houses have no knowledge of what’s expected before they can call a novel finished. They produce a first draft in a couple of months, follow Stephen King’s advice and set the book aside for a few weeks, then come back to it for what amounts to little more than a spelling and grammar check.

It’s not really their fault. Nobody told them that this was not how it was supposed to be. Nobody told them that they really did need to get a professional editor to work with them to produce a good, solid novel. The results are a published catalogue of half a dozen novels that read like middle of the road fan-fiction.

This is a style of writing I’m familiar with myself, having once had a passion for reading Stargate / Buffy crossover fiction. It’s a genre of writing that many writers excel at in their early writing careers, appreciating the immediate feedback from users, and the chapter by chapter publishing methods that are so common. Unfortunately, they often take this amateur approach with them as they venture into the more real world and self-publish their original works.

Finally, there’s cost. Professional editors are expensive. Good professional editors are more expensive still. In the world of old school publishing, this cost was and is borne by the publishing houses. In the new world of self-publishing, the cost belongs to the writer.

Many unpublished authors are not in a position to afford the costs of an editor, or convince themselves that they do not need to afford the cost. After all, they’re still learning. Three books down the road they’ll have improved, they’ll be selling more copies, and they’ll be able to afford that editor. Why not wait until then?

If a new writer can’t afford to pay to have their work professionally edited, if they feel that their only option is to learn on the job, then that is what they will continue to do. Wishing it were not so will not change things.

This is the backdrop to the development of SmartEdit.

SmartEdit was not designed to replace a human editor, it was created to offer the writer a first step when it comes to editing. With SmartEdit, they can make a start on the editing process themselves, improving their work as much as possible before deciding on a next step.

Should they also hire a professional editor? Of course.

But the better the writing, the less costly the editor. Most editors charge varying rates depending on the amount of work they will be required to do, and this depends on the quality of the writing offered to them.

SmartEdit can reduce this cost by helping the writer address many mistakes early. The software works best when it’s given a first draft of a novel to work with. The greater the material to draw upon, the more deductions it can make.

So how does it work?

SmartEdit applies up to twenty five individual checks to the writer’s work. It flags overused phrases, crutch words, clichés and redundancies. It builds list of adverbs and dialog tags in a novel, as well as an invaluable list of proper nouns. It monitors the beginnings of sentences, and allows the writer to identify an over reliance on a particular phrase or word. It separates dialog from the prose that surrounds it, allowing checks to be run on one or the other–an invaluable tool for monitoring over use of words and phrases in character dialog.

SmartEdit also identifies more mundane issues such as excessive punctuation, double spaces, conflicting straight and smart quotes, and en and em dash issues–the sort of thing that can easily slip past a human eye and goes unremarked upon by word processors such as Microsoft Word and Open Office.

But it’s still a tool. Calling SmartEdit a glorified spell checker would not be doing it an injustice. It’s just as true that a technically oriented writer could produce scripts to perform many of these tasks themselves, and chances are many do.

But for the vast majority of writers, seeing all this information in one place, sitting side by side with their work-in-progress, offers a real opportunity to make a solid start on the editing process, a process that is often foreign to a great number of self-published writers.

Many will balk at the suggestion that software has a place in the editing process. To which I would counter that it already does. Spell checkers, features such as Track Changes in modern word processors, not to mention the much derided grammar checker are already heavily used by writers.

How many writers perform a search in their word processor for words ending in ‘ly‘? This is a simple effort to seek out an over-reliance on adverbs, utilising the technology already built into their word processor. SmartEdit does it better, taking on the techie tasks and letting the writer focus on the writing.

So long as the software does not automate any changes or even casually suggest changes, then it has a productive use in an open minded writer’s toolbox. After all, it’s the result that counts–the polished, finely honed manuscript that a writer can truly be proud of.

The world of writing and publishing is changing. Those changes are well under way and visible to all. The world of editing will have to change to keep up. That’s where SmartEdit comes in.

02 Aug 2013

SmartEdit Version 3 — Coming Soon

Work on the next version of SmartEdit has been completed, and we’re delighted with the results. It is, without a doubt, a huge improvement over the current version, and we’re looking forward to getting it out there. Version 3 is due for release on Thursday, August 1st.

A little background: The commercial version of SmartEdit is seven months old. We’ve deliberately held off on any major changes during those months so that we could gather enough feedback from users, thereby ensuring that any changes introduced were the right changes, and not merely a quick response to the requests of a minority of users.

Over the past few months we’ve learned that most of SmartEdit’s users come from a Microsoft Word background, that most users are either published authors or authors who have finished at least one first draft of a novel, and that many of those users do not directly edit their work within SmartEdit, preferring to use the results of SmartEdit’s checks when editing in their favourite word processor — usually MS Word.

How has this impacted version 3?

We’ve redesigned the user interface from the ground up, bringing it more into line with MS Word, and other modern software. We’ve introduced a new, fuller featured word processor that can handle Word and Open Office files — no more saving to RTF or copying and pasting. We’ve made these changes so that users no longer need to bounce backwards and forwards between Word and SmartEdit. They can now complete their first draft in Word, open the same file in SmartEdit where they can feel comfortable making whatever changes are required, then move back to Word for the final manuscript preparation stage.

Here are a few screenshots of the new version:

Screenshot 1
Screenshot 2
Screenshot 3

What does this mean for existing users?

As the software is only seven months old, we’ll be allowing all current users to upgrade to version 3 at no cost. Once it’s available, you simply download and install over the existing version, and you’ll be good to go.

For new users, the price will be increasing on August 1st to $59.95. This is still a huge bargain, making SmartEdit one of the better priced editing products for writers on the market. Unlike many competitors, SmartEdit is not subscription based, so a single purchase is all that is necessary unless you wish to upgrade to a future version. If you are a new user, you should purchase before August 1st, to avail of the cheaper rate.

We hope you like the new version as much as we do.

21 Jul 2013

Self-Publishing & SmartEdit

The new respectability that is becoming attached to self-publishing on platforms such as Amazon’s KDP and Createspace is forever changing the publishing and writing industries. How it all ends up in five or ten years is still anybody’s guess.

My feeling is that we are at the very early stages of something new; that publishing and writing are about to be taken out of the hands of large corporations and gatekeepers, and placed firmly in the hands of individual writers and their readers. But this comes with a cost. While the sheer volume of good published writing is sure to increase enormously, so too will the amount of poorly written and poorly edited books.

This is already happening, especially in lucrative genres such as Romance and Erotica, where trying out a new author who has self-published a first or second book can lead to disappointment. You are often left feeling that the editing process has been skipped entirely, and that what you find yourself paying money for is little more than a first draft. Grammatical mistakes and poor sentence structure in the first chapter (sometimes even in the free sample), plot holes and clumsy prose on every second page, characters who change name halfway through a novel. None of this is uncommon in self-published work.

Where does SmartEdit come into all this?

It helps writers make a start on the editing process. For many writers, especially new writers who have just finished a first draft of their first novel, editing is a daunting task. Is it necessary? How much do I need to edit? Do I need a professional editor? How much do editors cost? Where do I start? All these questions are racing around in their minds.

SmartEdit answers the Where do I start? question. It runs a series of checks on a novel, looking for common mistakes and for areas that might need to be looked at in detail by the writer. The full results of SmartEdit’s checks can provide a writer with hundreds or thousands of possible areas for improvement in their novel. The writer can then spend days or weeks going through the results and making whatever changes they deem necessary.

The result: a tighter, leaner work, with fewer obvious mistakes and cringe worthy moments from readers.

But the process does not end there. The writer is not ready to publish just yet. SmartEdit is a first-pass editing tool. The novel still needs to be sent to a professional editor for fine tuning and to catch mistakes that automated software cannot catch.

So why use SmartEdit at all?

Editors cost money. And where once, that cost was borne by the publisher, for self-published writers the cost is theirs alone. Good editors charge varying rates based on a sample of the writer’s work. If there’s a lot of work to be done on a poorly edited manuscript, the cost goes up. If the work involves few obvious errors (the kind that SmartEdit helps identify), the cost often comes down.

SmartEdit should pay for itself when used on a single short story.

But cost is not the only reason to use SmartEdit. It helps writers improve their skills, helps them to identify mistakes that they make over and over: an embarrassing fondness for a particular adverb or dialog tag, a propensity to begin too many sentences in the same way, an over use of excessive punctuation when things get exciting.

The shift towards a respectable form of self-publishing and the sheer volume of writers who are embracing it has helped SmartEdit gain fast acceptance into the tool boxes of many writers. It can help you too.

24 May 2013

Extracting Character Names — How impossible it proved to be

The initial plan for version 2 of SmartEdit included extracting a list of character names and displaying them alongside all the dialog used by that character through an entire novel. This would have allowed the writer to easily check each individual character’s dialog as part of the editing process.

Sounds great, right?

If it had been possible to implement such a feature it would have been of huge benefit to writers, allowing them to monitor a character’s dialog over the course of an entire work — keeping a close watch on word usage to spot words, phrases and speech that didn’t fit the character, and allowing the writer to catch inconsistencies that are easy to miss when the dialog is read as part of a much larger work.

It wasn’t possible, and this surprised me. My early thoughts were that it would turn out to be a straight forward task. When we read a well written novel, there is rarely any doubt as to who is speaking. Dialog tags accompany most dialog (Sarah said, Mr. Smith replied, etc.), so how hard could it be to use those tags to build a dialog map for each character?

The problem I found was that in most novels, character names are rarely tied directly to dialog. Now and then, sure, but across the length of a novel they may only appear 5% of the time. The remaining 95% of that character’s dialog is attributed using “he said” or “she answered,” combinations of descriptive identifiers such as “her friend replied,” or no tags at all.

Once a well written conversation gets going, it’s not uncommon to read full pages of dialog where no actual character names are used. A sufficiently intelligent computer may eventually be able to work through this and put such a map together, but for version 2 of SmartEdit it was not possible.

10 May 2013

The History of SmartEdit

SmartEdit began as a small collection of features in PageFour about five years ago. At the time, PageFour was undergoing an identity crisis, and I was searching for a new direction to take the product.

Enter Smart Edit – a first stab at introducing editing capabilities into PageFour. Two features were added as part of the Smart Edit package: a word frequency counter and a phrase frequency counter. The first was a common feature in various software packages at the time, the second was unique – and incredibly cool and useful.

It failed to grab the interest of PageFour’s users. There was some interest, but not enough to warrant a shift in direction from writing into editing. This was back in 2008, well before self-publishing began to be seen as bordering on the respectable.

Its time hadn’t come.

In 2011 I was testing a new development environment, and decided to build a standalone SmartEdit product as part of the testing process, decoupled from PageFour and available as a free download.

The result was a simple word processor that used the word and phrase frequency counter of PageFour. For a year or so, it had moderate interest, with a few downloads a week, but little exposure. There were dedicated followers who wanted more features and were prepared to pay for them, but the number was small.

Then something happened. In mid 2012, the free version of SmartEdit was picked up by various social media sites and online writing magazines and forums. Bloggers started blogging about it and tweeters started tweeting. It helped that it was free. People love the free stuff.

Kindle sales were booming. eBooks had arrived. Self-publishing was the new thing. The timing was right for a product like SmartEdit.

It was Summer of last year that I started work on a feature heavy version of SmartEdit. The first version was ready in December 2012, and sales were immediate. The initial version wasn’t perfect, and it still isn’t. There are missing features and things that need improvement, but the feedback so far has been great. It’s almost four months later, and sales continue to improve month by month.

SmartEdit is filling a need amongst writers.

20 Apr 2013

Version 2.101 Released on 28th March

A new version of SmartEdit was released this afternoon (Irish time). This is the first new release since the licensed version was launched in December 2012, and contains a number of small improvements based on feedback from users.

Two areas of SmartEdit have been significantly improved: the Misused Words list and the Properties tab.

The Misused Words list was very much a work in progress (and still is). The first version returned far too many results, making it difficult to use. The purpose of the feature is to highlight words and phrases that are often misused in place of other words. Examples would be: complement and compliment, advice and advise, lead and led.

The first problem with the previous version was that it also included all to common words such as “their, there, they’re” and “to, too,” which caused the results list to balloon to unmanageable proportions, often concealing genuine problems behind a long list of commonly used words. The second problem was that both words in the word pairs were given equal weighting, when the reality is that usually, only one of the two words or phrases would commonly be misused.

Example: “should of” in place of “should have.”

I cringe whenever I read that phrase, or its all too common sister phrases “would of, could of.” The new version of SmartEdit raises each instance of “should of” as a problem, but ignores “should have,” as it’s rarely used in error.

The more commonly misspelled words such as “their, there, they’re” have been moved to the user defined Monitored Words List. This means you can still keep an eye on them – if you’re prone to those sorts of errors – but you can chose to exclude them from checks if you so wish.

The second major change to version 2.101 was to the Properties tab. The Properties tab lists general information about your work, such as word counts, smart and straight quite counts, as well as hyphen and dash counts. It’s especially useful to identify misused dashes or free floating straight quotes when you thought you had smart quotes switched on.

Each result now works as a link. Clicking on the link jumps to the first occurrence of the problem, allowing you to correct it straight away. Subsequent occurrences can be reached by using the SmartEdit Find dialog, which is automatically pre-filled with the character you’re looking for.

This is a major usability improvement.

Minor changes included in this release: a re-sorting of the sentence start list alphabetically, making it easier to come back to your editing at a later point and pick up where you left off. I expect this change to be rolled out to other results lists in the future.

The 10 Day trial has been reset for all users. This means that if you tried out SmartEdit back in December, you can now download the latest version, and your trial will restart, allowing you to test the software again.

Version 2.101 is free to all registered users. To install, close your current version of SmartEdit, then download and install the new version.

28 Mar 2013

Writers don’t need special software

In centuries past, books were hand copied by monks in distant lands, laboured over for months or years at a time, then read by a few dozen, or at most a few hundred people. Then Gutenberg came along, and before they knew what was happening, monks all over Europe were looking for jobs and the ordinary man on the street was reading the latest trashy novel.

But at least those first drafts were hand written, in whatever mangled scrawl the writer used. In many cases, hand written early drafts are still proudly on display in museums all over the world. And then the typewriter came along and made first drafts just another printed page.

Woe to writing!

The English author Kingsley Amis never used a word processor. He thought they were god awful machines that could never replace his trusty Adler typewriter that had served him well for thirty years.

It’s 2013, and we still hear variations of the monk’s and Kingsley’s laments from Luddites across the world. The only difference is the target of their ire. No one these days has a problem with printed books. No one wishes to go back to the oral tradition, before this pesky ‘writing’ took hold and changed storytelling forever. No one apart from those living in Germanic religious sects in the US wants to use a typewriter.

The word processor is the new all-you-need writer’s aid. It’s the quill and the typewriter of today, and Microsoft Word is the flagship. God help anyone who suggests that something new might be of use to writers.


Sure, writers today could write their novels by hand. They could use a typewriter. They could turn to figure painting and tell their story by adorning the walls of French caves with pictures of the Mars Rover.

But why?

Specialised technology exists in every field. A soccer Mom drives a different car to a salesman, who drives a different car to a policeman and an eco-warrior. Travel to any large city by train, and chances are you’ll take an intercity powered by petroleum of some kind to reach the city, a light rail system run on electricity to reach your destination suburb, and a tram or bus to reach the street or building you’re heading for. Specialised technology to meet special needs. People who live different lives want different cars. Different types of travel are best served by different types of transport.

Writing is no different.

A journalist wants one thing from his writing software, a student writing an essay for college wants something a little different, the guy putting a resumé together something else entirely.

And the novelist, the poet, the screenplay writer?

They want something just that little bit unique too. And why not? It wasn’t the Adler typewriter that made Kingsley Amis a great writer. The spell checker in Microsoft Word didn’t turn mediocre writers into Pulitzer or Booker winners. The tool itself doesn’t improve writing, all it does is make the working life of the individual writer that little bit easier and more productive.

The creative stuff, that thing that great books are made of, comes from inside the writer’s head. Writers may not need special software, but why shouldn’t they have it?

27 Mar 2013

« Previous Page