Skip to content

productivity

Migrating Modules/Directories Between GIT Repos

Disclaimer: I am not very good at using Git. The vast majority of time I just need to use a tiny subset of its features, and most likely my usage of Git resembles how I was using SVN in the past. Hence this post is actually a collection of resources I found online, which I document here for my own future benefit.

Working in large projects many times it means you have to move code around in different repos. For example, in many cases developers start building an application, maintaining all the code in one repo, but as this application is getting mature, or as parts of the app have been extracted in 'framework' like code there is the need to move those parts in a new repository. It is always beneficial to maintain the history of those modules/directories/file.

I came across such a situation recently and below are the steps I followed to achieve that.

Setup

For clarity, let's assume our setup is as below:

  • SourceRepo is the name of the repository that our code that we want to migrate lives at
  • Dir_I, Dir_II, Dir_III are the directories (and the files, other directories inside those) from the SourceRepo that we need to migrate
  • TargetRepo is the new repository, which exists and we need to migrate the above 3 directories, including their history

Steps

  • Clone the SourceRepo
git clone git@github.com:username/SourceRepo.git
cd SourceRepo
  • Remove the remote origin

This is optional and mainly done as a precaution, so that nothing will be pushed to remote origin by accident

git remote rm origin
  • Filter the commits in all branches that related to the diretories we want to keep

This step will blow away every other directory in the SourceRepo, only keeping the above 3 directories, along with their history.

git filter-branch --index-filter "git rm --cached --quiet -r --ignore-unmatch -- . && git reset $GIT_COMMIT -- Dir_I Dir_II Dir_III" --tag-name-filter cat -- -- all

This step will result in the final form of SourceRepo.

  • Clone the TargetRepo
git clone git@github.com:username/TargetRepo.git
cd TargetRepo
  • Add the SourceRepo as a remote to the TargetRepo
git remote add localSourceRepo ../SourceRepo
  • Fetch the index of the newly added remote localSourceRepo
git fetch localSourceRepo
  • Create a branch out of localSourceRepo

This branch will effectively have all the directories along with their old Git history, that was kept from SourceRepo

git branch temp remotes/localSourceRepo/master
  • Create a branch out of TargetRepo master

This is an optional step, but will help us create an intermediary branch where we will merge changes from SourceRepo and TargetRepo. Once the merging and any conflict resolution is completed (if any), someone can raise a PR to merge this to master

git checkout -b fromMaster
  • Merge SourceRepo branch to TargetRepo branch

In some cases (i.e. you had an identical named dir in TargetRepo) conflicts will arise. Those conflicts can be resolved in this step.

git merge temp
  • Once happy with the merging, a PR can be raised to merge fromMaster branch to master
  • Cleanup

This is an optional step. Most likely someone will be deleting all those temporary clones and dirs that were created. But for the benefit of the reader, we no longer need the localSourceRepo remote that we have added and also the temp and fromMaster (considering it was merged already) brances.

git remote rm localSourceRepo
git branch -D temp
git branch -D fromMaster

Resources

Managing JDKs in MacOS

With the increasing number of JDK builds and the more frequent release candence, I found it hard to keep track what I had installed in my MacOS and switch between them on the fly.

Even in 2019 my preferred version of Java is 1.8, probably because this is the version I am using at my work. But depending on the occasion I find myself experimenting with newer features from later versions, or even from experimental builds:

  • JShell from Java 9 onwards
  • EpsilonGC
  • The use of var since Java 10
  • Value types in Project Valhalla builds
  • etc…

In addition, nowadays on my personal computer I mainly use Java builds from the AdoptOpenJDK project. But there are other builds which I have installed on my MacOS to try out:

Hence I spent some putting together some bash functions that give me a hand managing and switching between those versions.

For the complete bash script see here, but the highlights are:

  • List JKDs
  • List Different JDK builds
  • Set JDK to a specific version/vendor

Note that java_home=/usr/libexec/java_home

Identifying a high-CPU Java Thread

High CPU utilization Java application

Every now and then we find ourselves in situations when a single Java process is consuming a high percentage of CPU.

After investigating and ruling out high CPU because of continuous GC cycles or other pathogenic reasons, we find ourselves in a situation that we need to identify the business logic that causes those CPU spikes. An easy way of doing so is try to identify the thread(s) that is consuming most of the CPU and try to pinpoint the caveat.

There are a few utilities (i.e. top, htop) that let us see a process as a tree along with the threads that live inside that process' space. After identifying the thread's ID, it is pretty easy to translate the ID to its HEX value and identify the actual thread in a Java application (i.e. by taking a thread dump).

Example

As an example the following Java program, uses two application thread's (main thread and a thread created by the user), one thread is spinning forever generating random values. The main thread occasionally, reads those random values.

https://github.com/nikkatsa/nk-playground/blob/master/nk-dummies/src/main/java/com/nikoskatsanos/spinningthread/SpinningThread.java

It is expected that this would be a high CPU utilization application (see above image).

Find the Rogue Thread

After identifying the Java program's PID (i.e. with jps or something like ps, top, htop), we can run an application like htop as below

htop -p${PID}

A user can view that isolated process along with its threads. Usually htop would show user space threads by default, but if not is easy to do by going to the setup page and selecting the appropriate option on Setup -> Display Options.

Then a user should see an image like the below.

That shows the application's PID along with its threads, reporting the metrics (CPU, Memory etc) for each thread. From there someone can easily identify that thread 12820 is consuming a great percentage of CPU, hence it should be our caveat.

Translating Thread's ID to HEX

The next step would be to translate that thread's decimal ID to its HEX value, which is: 0x3214

Getting a thread dump

Knowing the thread's HEX value, the user can take a thread dump and easily locate the thread and its stack trace.

Full thread dump Java HotSpot(TM) Client VM (25.65-b01 mixed mode):

"Attach Listener" #8 daemon prio=9 os_prio=0 tid=0x64900800 nid=0x3340 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Spinner" #7 daemon prio=5 os_prio=0 tid=0x6442a800 nid=0x3214 runnable [0x6467d000]
   java.lang.Thread.State: RUNNABLE
        at java.util.concurrent.ThreadLocalRandom.nextDouble(ThreadLocalRandom.java:442)
        at com.nikoskatsanos.spinningthread.SpinningThread.spin(SpinningThread.java:16)
        at com.nikoskatsanos.spinningthread.SpinningThread$$Lambda$1/28014437.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - <0x659c2198> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"Service Thread" #6 daemon prio=9 os_prio=0 tid=0x76183c00 nid=0x3212 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"C1 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x76180c00 nid=0x3211 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7617f000 nid=0x3210 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x76162000 nid=0x320f in Object.wait() [0x64f9c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x65806400> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x65806400> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

   Locked ownable synchronizers:
        - None
"C1 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x76180c00 nid=0x3211 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x7617f000 nid=0x3210 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
        - None

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x76162000 nid=0x320f in Object.wait() [0x64f9c000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x65806400> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        - locked <0x65806400> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

   Locked ownable synchronizers:
        - None

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x76160800 nid=0x320e in Object.wait() [0x64fec000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x65805ef8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
        - locked <0x65805ef8> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
        - None

"main" #1 prio=5 os_prio=0 tid=0x76107400 nid=0x320c waiting on condition [0x762b1000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at java.lang.Thread.sleep(Thread.java:340)
        at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386)
        at com.nikoskatsanos.spinningthread.SpinningThread.main(SpinningThread.java:40)

   Locked ownable synchronizers:
        - None

"VM Thread" os_prio=0 tid=0x7615d400 nid=0x320d runnable

"VM Periodic Task Thread" os_prio=0 tid=0x76185c00 nid=0x3213 waiting on condition

JNI global references: 310

The nid value (nid=0x3214) should match the HEX value of the thread's decimal ID

As seen, in the above case is obvious that thread with name 'Spinner' is the high CPU utilization thread we are looking for. After this point the user can investigate the application's logic and determine the root cause.

Introduction

I use Vim quite often in my day to day activities. I do not use it to write code on it, but I tend to find myself logged into a linux host, where I need to edit or update files. Vim is my editor of choice for doing that.

I am not really an expert on Vim, and I tend to use a particular set of commands, hence I decided to summarize the most common commands and shortcuts I use and have this blog post as a cheatsheet whenever I want to refresh my memory.

VIM Command Categories

I tried to group the commands I use into categories like Navigation, Edit, Search and Options.

Navigating in VIM can be done using the arrows. This navigates the cursor wherever the user wants. However, as VIM is really optimized to do everything with keyboard, in the most efficient way, navigation can also be achieved with keys 'h', 'j', 'k', 'l'. Those keys map nicely into the user's fingers, without the need to move your hand to reach the arrows.

Another very common function I find myself doing is going to lines. Someone can switch on line number by :set numbers (see options section). Moving directly to a line is as simple as ${lineNumber}G, which can be read as 'go to line ${lineNumber}'.

Additionally moving to absolute positions like the file's start or end, or to the begginning or end of a line is also fairly common.

Moving one word at a time, or a continuous block (like a camel case word) can be achieved with 'w' and 'W' respectively

Edit

I have grouped actions like inserting characters, deleting character, copying and pasting under the Edit category

Operations like copying and pasting are fairly common when editing files on a unix host

Search commands are very useful either in VIM or in less. Fortunately enough both tools have the same mechanism for searching

VIM can highlight any found search terms by setting the option :set hlsearch (see options section)

Options

VIM is highly customizable. User can set a profile with the desired properties and VIM behaviour at .vimrc file. The most common options I personally used are the below.

Java 9 Process API

In a previous blog post I wrote about one of my favourite features of Java 9, the JShell. At this post, I will write about another feature I am excited about. The new Java 9 Process API. I will also present some code showing how powerful and intuitive it is.

The new API adds greater flexibility to spawning, identifying and managing processes. As an example, before Java 9 someone would need to do the following in order to retrieve the PID of a running process:

The above is not intuitive and seems like a hack. It feels to someone that the Java process should at least easily expose its own PID.

Moreover, I quite a few times needed to spawn new child processes from inside a Java process and manage them. The process of doing so is very cumbersome. A reference to the child process has to be kept throughout the program's execution if the developer wishes to destroy that process later. Not to mention that getting the PIDs of the children processes is also a pain.

Fortunately, Java 9 comes to fix those issues and provide a clean API for interaction with processes. More specifically two new interfaces has been added to the JDK:

1. java.lang.ProcessHandle 2. java.lang.ProcessHandle.Info

The two new interfaces add quite a few methods. The first one methods for retrieving a PID, all the processes running in the system and also methods for relationships between processes. The second one mainly provides meta information about the process.

As someone would expect most of the methods have native, platform specific implementations. The OpenJDK's implementation of ProcessHandle can be found here. Also the Unix specific implementation can be seen here.

I have created a very simple program which makes use of most of the features of this new Process API. The program does the below:

  • Can retrieve the running process' PID
  • Can start a long running process
  • Can start a short running process, which terminates about ~5seconds after starting
  • Can list all child processes that were spawned by the parent one
  • Can kill all child processes that were spawned by the parent one
  • Attaches a callback when a child process exits. This is done using the onExit() method of the ProcessHandle

The sample class is provided below. For the entire example please see here:

JShell

As of now, Java 9 official release date is 27.07.2017. According to openJDK mailing list the push back was due to the most anticipated feature of Java 9, which is the modularisation of the JDK or commonly known as Project Jigsaw.

I am not as much excited for this feature as I am for the brand new JShell. Many people criticise the language's verbosity and sometimes the amount of code that is required to do some stuff. I do not disagree that this, in many cases, is true. But, what I was missing mainly from Java was the ability to quickly evaluate an expression/algorithm/piece of code.

For example, many times I find myself needing to try something quick which involves reading a file or reading something from the web and performing some manipulation on it. Or even sometimes testing out a lambda expression to see its behaviour. Up to this point, actions like that were a bit cumbersome, as it involved the creation of a class, a main method and the execution of that program.

JShell is introduced to solve problems like that and more. Also known as Project Kulla JShell is an extremely useful tool. It is a REPL (Read Evaluate Print Loop) tool. Similar ones exist in various other languages like Python, Perl, even Scala.

For someone to use JShell she/he needs to download JDK 9. Then all she/he has to do is to navigate to /bin directory and execute the jshell command.

Firstly, the JShell itself prompts the user to type /help intro

The jshell comes with auto-completion features, so the user can press Tab and see a list of commands depending on the first letter she/he typed:

A list of help command can appear on the output by typing /help.

By default JShell has to import the classes that the user is going to use. It comes with a pre-defined set of common classes already imported:

A user can import any JDK class, or even her/his own classes by adding to the classpath:

As someone can notice it is not mandatory to add semicolons in the end of statements. However, it is mandatory to add them if the user adds a class or a method.

Each expression the user writes on the console is evaluated and printed on the standard output. If an expression has a return type that return type is automatically assigned in a variable that the shell creates on the fly. Of course, later on the user can make use of that variable as normal:

User can define methods outside of classes. Additionally, classes can be defined and referenced as normal:

A very nice feature is the fact that the user does not need any try{}catch{} blocks for methods which define checked exceptions:

Finally, the user can see the defined methods, types and variables and reset her/his session:

Concluding, I believe JShell will be a nice to have tool. By exploring it I am pretty sure people will come up with some interesting uses of it.

IntelliJ Live Templates

I am a huge fan of IDEs. I truly believe that a powerful IDE can boost a programmer's productivity by orders of magnitude. Additionally, i also believe that each developer should treat his/her IDE as a tool in his/her toolbox and try to get the most out of it. Having said that, making ourselves familiar with the IDE's features (i.e debugging, profiling, refactoring utilities) and shortcut keys is a must. Everybody should invest time in those things. In the beginning it may be tedious, but in the long run it really worths it.

My personal choice of IDE is IntelliJ IDEA from Jetbrains. The Jetbrains guys have done an amazing job in creating one of the best technological products out there, according to myself. Their offerings are not only Java related for those who are interested in checking their products.

IntelliJ IDEA is such a powerful IDE that a developer can do almost everything from the keyboard. When i started using IntelliJ i forced myself to learn every shortcut key that i was using and just use that. In some cases i disconnected my mouse so that i would be forced to just type my way in achieving things. And i found out that i could work on it as comfortably with or without a mouse. I also found out that just by using the keyboard i was more productive, as my 'muscle memory' was kicking in every time and i did not have to take my hand off the keyboard to reach the mouse and think of the action that i wanted to perform, being in danger of loosing my momentum.

One of the features that i love and i use is their "Live Templates". The live templates is a mechanism where the developer can use already existing or create his/her own code snippets and add those to code by just typing an abbreviation followed by the 'tab' key. Someone can create a new live template or a new live template group by going to Preferences -> Editor -> Live Templates. It definitely makes sense to have the most common patters as live templates. Three of my favourite that i use almost always when i program are the following.

Unit test case: intelliJTestLiveTemplate

Main method: intelliJMainMethodLiveTemplate

SLF4J Logger: intelliJSLF4JLiveTemplate intelliJSLF4JLiveTemplateII

There are lots of others built into the IDE, like for-loops, System.out.println() statements etc, that are commonly used.  I believe that there is no limitation on any live template that someone want to create according to his/her needs.

But whatever tool you use in your everyday life, i would strongly recommend spending some time in learning that tool inside-out, as the benefits of that are tremendous.