I have recently started implementing different distributed system protocols to get some understanding how they work. I think that using Akka Actors to simulate hosts is a good choice because they are easy to set up. What is more you can kill actors on demand to test some failure scenarios.
First protocol I implemented is Paxos - probably the most important consensus protocol, which guarantees storng consistecy, full transactions, read/write failover and prevents from data loss. This comes with the cost of higher latency and lower throughput. In terms of CAP theorem, Paxos is used to make your system CP. I used this sweet article as a reference.
Implementation can be found on my github repository. Go to the Test.scala and run some simulations.
-
When I started working as a Java Developer, me and my teammate got a first task to repair all broken tests (great task for new starters!) in some old project. Replacing some old configuration and upgrading a few libraries helped making the tests status green but there was another problem. End-to-end test suite took a few hours to pass making it impossible to get the feedback for a new change quickly. Sometimes build run for a few hours and hanged so E2E tetst became excluded from the feedback loop at the end.Most of the tests from that suite did the same thing:
- Invoke a few commands on the mainframe through HTTP interface
- Sleep for 30 seconds (probably just to be extra safe) for each command and check if the result was persisted to database by another system.
It turned out, that first try to reduce the sleep time to 15 seconds by my teammate, didn’t make any tests fail, so that was a nice first try. But what was the optimal wait time for each test?After some time, we tried another approach. If the flow has a asynchronous nature then why not testing it asynchronously.I introduced this simple class (or at least class which does the same):Method await checks some assertion in a loop until either it won’t throw AssertionError or the timeout is passed. This assertion is usually some state we are waiting for that should be ready in the nearest future. If it is not ready after the timeout, the last message from the AssertionError will be thrown wrapped as a RuntimeException. This greatly reduces test time because if the result is ready earlier, we don’t have to spend unnecessary time in the sleep method.
To represent the assertion result we can use classes (minimal implementation):We also need to prepare our AssertionResult suppliers - wrappers which fetch the data and invoke assertions on the result. Usually it is possible to define generic supplier if the way we fetch the data is similar. For example we can write this builder method:We are ready to replace Thread.sleep() with call to Await in our tests:Conclusion
Don’t overuse Thread.sleep() in your test code. According to this Google research it is one of the main reasons of flaky tests. Asynchronous approach may complicate your tests but the time savings can be tremendous. If you are looking for some existing implementations there is a nice class in Spock which does the same.0Add a comment
-
In this article I am going to share some cool features I stumbled upon while coding with Intellij. These are not the most popular/productivity improving ones - for these you should watch this video.
1. Set debugger breakpoint by pattern
Problem: You want to check if the debugger steps into some code during the run. Usually you suspect where the flow will end up but sometimes finding the exact line could be hard.Solution: Breakpoints in Debugger pane -> Click (+) -> Java Method Breakpoint -> Fill class/method pattern.For example this setting will stop the debugger anytime the code enters any method from any class from com.fasterxml.jackson package.
2. Attaching custom log file to be followed after Spring Boot app startup.
Problem: You have your app log configured to write to the file after the startup. However, this makes it harder to read that log because now you have to open this file somehow instead of just checking the console output.Solution: Run/Debug Configuration -> Your Spring Boot app -> Logs -> Click (+) -> Fill alias and Log File Location. This will open additional pane with file content after you run the configuration that is easy to work with.3. Invoke any code one the screen when in debug mode
Problem: You want to check the result of invoking some code on the screen but you are to lazy to paste it to Evaluate Expression console.Solution: When the program is paused in debug mode, hit the alt key and click any fragment you want to see the result.4. Compare with clipboard
Problem: You would like to compare some text the same way Intellij’s VCS diff shows changes in files.Solution: Copy text to the clipboard -> Select any other text -> Right Click -> Click Compare With Clipboard.I think it is worth to have a key assigned to this one.5. Shelve changes
Problem: I used to use Git stash command externally to save some changes I didn’t want to commit yet. Is there a way to do this from Intellij?Solution: VCS -> Local Changes -> Select the files you want to stash -> Right Click -> Shelve changes.You can move them back from the shelf by right clicking on the selection and choosing Unshelve.Conclusion
I hope you have learned at least one thing that might be helpful at you work. Please share other cool but less know Intellij features in the comment section.0Add a comment
-
I have recently pushed very simple Map Reduce concept implementation on my Github account (click). My idea was to focus on the concept and mock the rest. You can follow the code to understand how it works but I will enumerate most implementation details:
- Text files are represented as Strings and stored in memory
- Master is responsible for splitting the data and scheduling the map/reduce tasks to workers
- Workers are represented as threads, they are run using CompletableFuture api
- When map task is finished, the combine function will be run on the result
- When Master will notice that all map tasks have finished, it will take all resulted distinct keys and pass them as an argument to reduce tasks
- When reduce tasks are finished, result is printed and the executor is closed
It is easy to define new user programs, check example package.In memory data:
"file1.txt" -> "This is the first file \ncontent. " "file2.txt" -> "And this is the second file content. " "file3.txt" -> "More text in \nthird file" "file4.txt" -> "And some random text here" "file5.txt" -> "Why not \none more" "file6.txt" -> "Lululu tengo manzana"
Distributed Grep (search for "And"):
New map task for key: file5.txt New map task for key: file4.txt New map task for key: file3.txt New map task for key: file6.txt New map task for key: file1.txt New map task for key: file2.txt Combine task for: file2.txt Combine task for: file4.txt Writing: IntermediateResult{key=file2.txt, value=And this is the second file content. } Writing: IntermediateResult{key=file4.txt, value=And some random text here} New reduce task for: file4.txt New reduce task for: file2.txt Results: file4.txt - And some random text here file2.txt - And this is the second file content.
Word count:
New map task for key: file5.txt New map task for key: file4.txt New map task for key: file3.txt New map task for key: file6.txt New map task for key: file1.txt New map task for key: file2.txt Combine task for: not Combine task for: here Combine task for: the Combine task for: manzana Combine task for: the Combine task for: More Writing: IntermediateResult{key=More, value=1} Writing: IntermediateResult{key=the, value=1} Writing: IntermediateResult{key=here, value=1} Writing: IntermediateResult{key=the, value=1} Writing: IntermediateResult{key=not, value=1} Writing: IntermediateResult{key=manzana, value=1} Combine task for: file Combine task for: random Writing: IntermediateResult{key=file, value=1} Writing: IntermediateResult{key=random, value=1} Combine task for: file Combine task for: tengo Combine task for: file Combine task for: more Writing: IntermediateResult{key=tengo, value=1} Writing: IntermediateResult{key=file, value=1} Combine task for: some Combine task for: third Writing: IntermediateResult{key=some, value=1} Combine task for: This Combine task for: Lululu Writing: IntermediateResult{key=Lululu, value=1} Writing: IntermediateResult{key=more, value=1} Writing: IntermediateResult{key=file, value=1} Combine task for: one Writing: IntermediateResult{key=one, value=1} Writing: IntermediateResult{key=This, value=1} Combine task for: And Writing: IntermediateResult{key=And, value=1} Writing: IntermediateResult{key=third, value=1} Combine task for: in Combine task for: text Writing: IntermediateResult{key=in, value=1} Combine task for: content. Combine task for: Why Writing: IntermediateResult{key=content., value=1} Writing: IntermediateResult{key=Why, value=1} Combine task for: And Writing: IntermediateResult{key=And, value=1} Combine task for: this Combine task for: is Combine task for: text Writing: IntermediateResult{key=text, value=1} Writing: IntermediateResult{key=text, value=1} Writing: IntermediateResult{key=is, value=1} Writing: IntermediateResult{key=this, value=1} Combine task for: first Combine task for: content. Writing: IntermediateResult{key=first, value=1} Writing: IntermediateResult{key=content., value=1} Combine task for: is Writing: IntermediateResult{key=is, value=1} Combine task for: second Writing: IntermediateResult{key=second, value=1} New reduce task for: not New reduce task for: more New reduce task for: one New reduce task for: the New reduce task for: file New reduce task for: This New reduce task for: content. New reduce task for: first New reduce task for: Lululu New reduce task for: More New reduce task for: in New reduce task for: text New reduce task for: Why New reduce task for: random New reduce task for: here New reduce task for: third New reduce task for: tengo New reduce task for: manzana New reduce task for: is New reduce task for: second New reduce task for: this New reduce task for: And New reduce task for: some Results: not - 1 more - 1 one - 1 Why - 1 the - 2 file - 3 This - 1 content. - 2 is - 2 first - 1 manzana - 1 tengo - 1 Lululu - 1 More - 1 third - 1 in - 1 text - 2 here - 1 random - 1 some - 1 And - 2 this - 1 second - 10Add a comment
-
In this article I will try to map methods of Java’s Optional to Kotlin’ssimilar scattered language features and built-in functions. The code in the examples is written in Kotlin, because the language has all the JDK classes available.
Representation
Let’s start with the representation. The Opitonal usage, requires creating a new object for the wrapper every time some value is wrapped or transformed to another type, with exclusion of when the Optional is empty (singleton empty Optional is used). In Kotlin there is no additional overhead, lanuage uses plain old null. The code below shows both approaches:Map using method of inner value’s type
To transform the value inside Optional using the inner value’s method we can apply a method reference to map. To do the same in Kotlin, we can use safe call operator .? as demonstrated below:Map using external mapper
If the transformation cannot be performed by simple method call then Optional’smap method is happy to take lambda as well. Kotlin provides built-in method let, which we can invoke on any object. In our case we also need to use safe call operator to skip the calculation for null values:Filter
In Optional, filter allows to remove the value inside, if the provided predicate test returns false. Kotlin offers two built-in functions with this behaviour - takeIf and takeUntil. First is simialar to the Optional’sfilter, second one drops the value if the predicate returns true - opposite to takeIf. Example code:FlatMap
There is no built-in Kotlin function with the flatMap behaviour because it’s actually not neccassary. Nullability is not represented by a type and simple mapping turns out to be working fine.IsPresent
In Kotlin this check can be performed by a simple null comparision:It is worth to know, that after we do this, compiler is sure that the value is present and allows us to drop all required null checks in the further code (String? becomes String).Get
Both Optional and Kotin approaches discourage users from getting the inside value with the straight call, because it may cause NPE. Kotlin introduces rude !! operator, which you should use only as a last resort:OrElse/OrElseThrow
Kotlin introduces elvis operator ?: which allows to set the default value in case of null or throw an excepcion:Conclusion
I hope reading this article will help you laverage your Optional's experience to quickly learn Kotlin'snull safety features. I think it is worth to give Kotlin a try if only to expand your programming horizonts.0Add a comment
-
Have you ever scrolled someone’s code and bumped into this weird method called flatMap, not knowing what it actually does from the context? Or maybe you compared it with method map but didn’t really see much difference? If that is the case then this article is for you.flatMap is extremly usfull when you try to do proper functional programming. In Java it usually means using Streams and Optionals - concepts introduced in version 8. These two types can be thought of as some kind of wrapper over a type, which adds some extra behaviour - Stream<T> wrapps type T allowing to store any number of elements of type T inside, whereas Optional<T> wrapps some type T to implicitly say that the element of that type may or may not be present inside. Both of them share methods map and flatMap. Before we move on, I want to make sure you understand what the map method is doing. It basically allows you to apply the method to the element inside the wrapper and possibly change it’s type:After applying the function toLong to our wrappers, their inner types changed to the second type of the toLong signature (the result type).Let’s examine the function Long::parseLong. If we call it using a string that is not actually a valid long it will throw NumberFormatException. But what if Java designers decide to implement it so it returns Optional<Long> instead of just Long and removed the exception? Our code for Optional part would look like that:Wow, that is nasty! When we appliednew method to our wrapper, the inner type was changed from String to Optional<Long> (the result type of toLongOpt which we applied). We don’t really need to have a double Optional because just one is perfectly fine. Now, to get the value we need to extract it twice, not mentioning how it would look like when we want to map it again without unwrapping… To restore it to the single type we would need to write a method like this one:This method will flatten our Optional<Optional<T>> to Optional<T> without changing the inner value. The code will look like this:This is exactly what method flatMap is doing. It first applies the function returning another Optional to the inside object (if present) and then flattens the result before returning it, so you don’t have to do it yourself. This is how we can use it:For Stream, we can use it in the situation when the function we want to map our elements with, returns the Stream. (example signature: Function<String, Stream<Long>>).That’s it. Just remember that flatMap = map + flatten.If you want to dive into the topic read about the Functor and Monad concepts and the relationship between them.0Add a comment
Add a comment