Monday, October 3, 2016
How to Use Python to Find the Zipf Distribution of a Text File
You might be wondering about the term Zipf distribution. To understand what we mean by this term, we need to define Zipf's law first. Don't worry, I'll keep everything simple.
Zipf's Law
Zipf's law simply states that given some corpus (large and structured set of texts) of natural language utterances, the occurrence of the most frequent word will be approximately twice as often as the second most frequent word, three times as the third most frequent word, four times as the fourth most frequent word, and so forth.
Let's look at an example of that. If you look into the Brown Corpus of American English, you will notice that the most frequent word is the (69,971 occurrences). If we look into the second most frequent word, that is of, we will notice that it occurs 36,411 times.
The word the accounts for around 7% of the Brown Corpus words (69,971 of slightly over 1 million words). If we come to the word of, we will notice that it accounts for around 3.6% of the corpus (around half of the). Thus, we can notice that Zipf's law applies to this situation.
Thus, Zipf's law is trying to tell us that a small number of items usually account for the bulk of activities we observe. For instance, a small number of diseases (cancer, cardiovascular diseases) account for the bulk of deaths. This also applies to words that account for the bulk of all word occurrences in literature, and many other examples in our lives.
Data Preparation
Before moving forward, let me refer you to the data we will be using to experiment with in our tutorial. Our data this time will be from the National Library of Medicine. We will be downloading what's called a MeSH (Medical Subject Heading) ASCII file, from here. In particular, d2016.bin (28 MB).
I will not go into detail in describing this file since it is beyond the scope of this tutorial, and we just need it to experiment with our code.
Building the Program
After you have downloaded the data in the above section, let's now start building our Python script that will find the Zipf's distribution of the data in d2016.bin.
The first normal step to perform is to open the file:
open_file = open('d2016.bin', 'r')
In order to carry out the necessary operations on the bin file, we need to load the file in a string variable. This can be simply achieved using the read() function, as follows:
file_to_string = open_file.read()
Since we will be looking for some pattern (i.e. words), regular expressions come into play. We will thus be making use of Python's re module.
At this point we have already read the bin file and loaded its content in a string variable. Finding the Zipf's distribution means finding the frequency of occurrence of words in the bin file. The regular expression will thus be used to locate the words in the file.
The method we will be using to make such a match is the findall() method. As mentioned in the re module documentation about findall(), the method will:
Return all non-overlapping matches of pattern in string, as a list of strings. The string is scanned left-to-right, and matches are returned in the order found. If one or more groups are present in the pattern, return a list of groups; this will be a list of tuples if the pattern has more than one group. Empty matches are included in the result unless they touch the beginning of another match.
What we want to do is write a regular expression that will locate all the individual words in the text string variable. The regular expression that can perform this task is:
\b[A-Za-z][a-z]{2,10}\b
where \b is an anchor for word boundaries. In Python, this can be represented as follows:
words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)
This regular expression is basically telling us to find all the words that start with a letter (upper-case or lower-case) and followed by a sequence of letters which consist of at least 2 characters and no more than 9 characters. In other words, the size of the words that will be included in the output will range from 3 to 10 characters long.
We can now run a loop which aims at calculating the frequency of occurrence of each word:
for word in words:
count = frequency.get(word,0)
frequency[word] = count + 1
Here, if the word is not found yet in the list of words, instead of raising a KeyError, the default value 0 is returned. Otherwise, count is incremented by 1, representing the number of times the word has occurred in the list so far.
Finally, we will print the key-value pair of the dictionary, showing the word (key) and the number of times it appeared in the list (value):
for key, value in reversed(sorted(frequency.items(), key = itemgetter(1))):
print key, value
This part sorted(frequency.items(), key = itemgetter(1)) sorts the output by value in ascending order, that is, it shows the words from the least frequent occurrence to the most frequent occurrence. In order to list the most frequent words at the beginning, we use the reversed() method.
Putting It All Together
After going through the different building blocks of the program, let's see how it all looks together:
import re
from operator import itemgetter
frequency = {}
open_file = open('d2016.bin', 'r')
file_to_string = open_file.read()
words = re.findall(r'(\b[A-Za-z][a-z]{2,9}\b)', file_to_string)
for word in words:
count = frequency.get(word,0)
frequency[word] = count + 1
for key, value in reversed(sorted(frequency.items(), key = itemgetter(1))):
print key, value
I will show here the first ten words and their frequencies returned by the program:
the 42602 abcdef 31913 and 30699 abbcdef 27016 was 17430 see 16189 with 14380 under 13127 for 9767 abcdefv 8694
From this Zipf distribution, we can validate Zipf's law in that some words (high-frequency words) represent the bulk of words, such as we can see above the, and, was, for. This also applies to the sequences abcdef, abbcdef, and abcdefv which are highly frequent letter sequences that have some meaning particular to this file.
Conclusion
In this tutorial, we have seen how Python makes it easy to work with statistical concepts such as Zipf's law. Python comes in very handy in particular when working with large text files, which would require a lot of time and effort if we were to find Zipf's distribution manually. As we saw, we were able to quickly load, parse, and find the Zipf's distribution of a file of size 28 MB. Let alone the simplicity in sorting the output thanks to Python's dictionaries.
CSS Grid Layout: Going Responsive
Throughout this series we’ve become familiar with Grid syntax, learned about some efficient ways of laying out elements on a page, and said goodbye to some old habits. In this tutorial we’re going to apply all of that to some practical responsive web design.
Media Queries
Let’s use the demo from where we left off last time.
It comprises two declared grids; our main grid and the nested grid within one of our items. We can control when these grids come into effect using media queries, meaning we can completely redefine our layout at different viewport widths.
Begin by duplicating the first grid declaration, and wrapping the duplicate in a mobile-first media query (I’m using 500px as the breakpoint, but that’s completely arbitrary):
.grid-1 {
/* grid declaration styles */
}
@media only screen and (min-width: 500px) {
.grid-1 {
/* grid declaration styles */
}
}
Now, within the first declaration we’ll change how our grid is defined, placing the whole thing in a single column. We’ll list just one column in our grid-template-columns rule, make sure the four rows we now have are defined with grid-template-rows, and arrange the layout with grid-template-areas:
.grid-1 {
display: grid;
width: 100%;
margin: 0 auto;
grid-template-columns: 1fr;
grid-template-rows: 80px auto auto 80px;
grid-gap: 10px;
grid-template-areas: "header"
"main"
"sidebar"
"footer";
}
We’ve also made our grid-gap just 10px by default, to account for smaller screens.
Here’s what that gives us. You’ll notice that we’re also using our media query to change the padding and font-size on our .grid-1 div items.
Our Nested Grid
That takes care of the main layout, but we still have the nested grid which remains stubbornly in its two column layout under all circumstances. To fix that we’ll do exactly the same as before, but using a different breakpoint to suggest a content-first approach:
.item-2 {
/* grid declaration styles */
}
@media only screen and (min-width: 600px) {
.item-2 {
/* grid declaration styles */
}
}
Check out the end result on CodePen.
A couple of things to note here:
- As we’ve said before, you can visually arrange grid items irrespective of the source order, and media queries mean we can have different visual orders for different screen widths. However, nesting has to remain true to the source; our nested grid items must always be visually and actually descendants of their parent.
- CSS transitions don’t have any influence over Grid layout. When our media queries kick in, and our grid areas move to their new positions, you won’t be able to ease them into place.
auto-fill and minmax()
Another (sort of) responsive approach to Grid is well suited to masonry-type layouts; blocks which size and flow automatically, depending on the viewport.
auto-fill
Our approach up until now has been to dictate how many tracks there are and watch the items fit accordingly. That’s what is happening in this demo; we have grid-template-columns: repeat(4, 1fr); which says “create four columns, and make each one a single fraction unit wide”.
With the auto-fill keyword we can dictate how wide our tracks are and let Grid figure out how many will fit in the available space. In this demo we’ve used grid-template-columns: repeat(auto-fill, 9em); which says “make the columns 9em wide each, and fit as many as you can into the grid container”.
Note: this also takes our gutters, the grid-gap, into account.
The container in these demos has a dark background to show clearly how much space is available, and you’ll see that it hasn’t been completely filled in the last example. So how do we do that?
minmax()
The minmax() function allows us to set a minimum and a maximum size for a track, enabling Grid to work within them. For example we could setup three columns, the first two being 1fr wide, the last being a maximum of 1fr, but shrinking no smaller than 160px:
grid-template-columns: 1fr 1fr minmax(160px, 1fr);
All the columns will shrink as you squish the window, but the last column will only be pushed so far. Take a look.
Back to our auto-fill demo, if we were to change our column width for minmax(9em, 1fr) Grid would place as many 9em tracks as possible, but then expand them to a maximum of 1fr until the container is filled:
Caveat: Grid will recalculate the tracks upon page reload (try squishing the browser window and hitting refresh) but it won’t do so on window resize. Media queries can be used to alter the values, but they still won’t play nice with window resize.
Conclusion
Let’s wrap up with some bullets:
- Media queries can help us completely rearrange Grid layouts by redefining
grid-template-areas(and other things) for different scenarios. - CSS transitions don’t have any effect on changes made to the grid layout.
- The
auto-fillkeyword is useful for filling up grid containers. - The
minmax()function complementsauto-fillnicely, making sure containers are properly filled, but doesn’t give us “responsiveness” in the true sense of the word.
With the lessons learned in this series, you’re armed to go out and start playing with Grid! Stay tuned for more Grid tutorials, practical exercises, solutions to common layout problems, and updates.
Useful Resources
- Rachel Andrew’s Grid by Example 29: minmax() and spanning columns and rows
- Video: Rachel Andrew (obviously) demonstrating minmax() on the Tuts+ homepage layout
- W3C Editor’s Draft: auto-fill
- W3C Editor’s Draft: minmax()
