How to simulate mouse movement. Move Mouse - simulating user activity
Very often, when working on a computer, you have to perform a number of similar actions with a mouse, keyboard, etc. All this is exhausting and smart programmers (and they are always smart!), to make the user’s work easier, they wrote programs called autoclickers. Let's look at how you can avoid routine work using the example of the wonderful eMouse program.
eMouse
We go to the site and its with. Installation is not difficult. After installation, we launch it and a window appears. 
On the left is the “Rec” (record) button - after clicking, your actions with the mouse and keyboard will be recorded. “Play” will play (repeat the recorded actions). And how many times can be set in the “x” field (in the picture it is set 1 time), and put a tick in the “loop Playback” field
At the bottom left are the Record: Keys and Mouse checkboxes, i.e. if these checkboxes are checked, then mouse and keyboard actions will be recorded (and if they are not checked, then accordingly).
You can view (and change) the settings. Click on the “Edit” link and context menu select “Settings”
Here are the settings.
Hotkeys installed. To start recording actions - F5, to pause - F11, to play - F12.
Recorded actions (script) can be saved, i.e. you can write a bunch of different sequences and call them as needed. 
Open “File” and select “Save Script as...” and save it where we need it.
AutoClicker (the most primitive)
If you need to do a huge amount of clicking with the right or left mouse button, then this program is for you. It's great to use VKontakte in PROFIT games! Super Clicker or Clicker. 
The program is paid, but its “free” version can be downloaded from. Let's launch it. The program interface is quite simple. 
How to work with the left mouse button is written on the left side, and how to work with the right mouse button - on the right (we'll deal with the sliders later). We use hotkeys.
We launch the program, i.e. Press ALT+1 on the keyboard. This does not mean that we press the ALT key on the keyboard, then the “+” key, and then the key with the number “1!. No, we just press the “ALT” and “1” keys at the same time.
We go to VKontakte, look for a game where you need to make a bunch of clicks, move the mouse and press ALT+1. And until we press ALT+2, the clicks will continue.
Ghost Mouse can be downloaded from . Installation is not difficult. Let's launch. 
To record actions, press the red button. Or press F9. To play the actions, press “Ctrl+Q“.
To begin with, you need to understand that people who are interested in automation programs various actions on the computer, simulating mouse movements and pressing keyboard keys, they don’t quite understand what they want. From the outside, everything seems simple - I need a program that will reproduce my actions with the mouse and keyboard, simulating them. Okay, let's try to understand the essence of the situation...
Macros - if you are looking for programs to automate and simulate actions on a computer, then you should know that there is such a thing as a macro. It's best to look at an example package Microsoft Office. Their programs (Word, Excel...) contain a built-in language editor Visual Basic. Basic, Visual Basic, vbscript, it doesn’t matter, the main thing is that it is a programming language related to Microsoft. Actually, a similar or analogous language syntax can be used in other programming languages from Uncle Bill. Unlike individual packages, Visual editor Basic in Microsoft Office is aimed at executing program code in an office environment, such as Word or Excel. There are similar tools in . So, you can write program code in the editor, save it, where it is called a macro, and run it periodically. A macro usually contains program code that allows you to quickly perform routine work, for example:
- Apply a style to every second paragraph.
- Convert all tables to text and apply the specified formatting.
- Automatically split contents and save into separate documents.
The macros themselves in other programs allow you to:
- Simulates a mouse button press.
- Simulation of pressing keyboard keys.
- simulate mouse clicks.
- Keyboard imitation.
- imitation of mouse movement.
Examples of various macros for Word and Excel a large number of. However, their main advantage is that they can be created without having to write code, you can simply press the PLAY button and your actions will begin recording (selecting text, applying a style). After finishing recording (by pressing STOP), a new macro will automatically appear with program code. Next, you can run the saved macro, which will work to automate actions. It is clear that indirectly, macros can also be considered unique programs for automating user actions in Word editor or Excel spreadsheet on your computer. Formally, macros are also created in Office, only within the editor.
You need to understand that such a concept as Macro is used in other areas of programming and languages.
Event interception– Aspects such as mouse movement and button clicks are usually called events. There can be many events, plus there are also various objects (windows, applications, window areas, and so on). Most programs for automating various user actions on the computer must be able to intercept events in the form of mouse movements, clicks and keyboard keystrokes, and then create their simulation. Some automation programs allow you to simulate or program mouse movements, others allow you to intercept, or vice versa, simulate keystrokes on the keyboard.
Making a program that can reproduce all your mouse and keyboard actions is not a problem; you can even turn on window processing. However, this will be just a simple simulation, which is suitable if you need to quickly make a few mouse clicks at one or more specified points, or automatically fill out several forms with data.
It is clear that there are many users, and there are also many tasks. Someone needs to automate the process of sorting and renaming files or directories. Someone wants to immediately apply one design to a hundred Word documents. In the latter case, for txt documents, you can use the Notepad++ editor.
Okay, let's look at a few simple programs on the topic of the article.
Macro Dollar program to simulate keyboard, mouse, movements and clicks
Macro Dollar- it's old and simple software automation and simulation of mouse and keyboard actions. Considering the fact that the program allows you to completely record and then simulate the actions of the mouse and keyboard, it can be called a full-fledged macro.
Having launched Macro Dollar, we will see a small window with buttons to start recording, stop, and play. There are also a few things to check off:
- Topmost– the program window will always be in the foreground.
- Loop– if you check this option, the process of playing the mouse and keyboard simulation will continue endlessly.
- Shake- “shaking” function, when the mouse cursor shakes while playing a recording.
- Play or record iconized– the program window is minimized during recording.
Judging by the icons and text field, the program allows you to create your own macros using commands written in text scripts. However, it’s enough for me that Macro Dollar can record mouse and keyboard actions and then simulate them.
Macro Dollar is a portable program.
WinMacro is another one simple program simulating mouse movements and keyboard buttons
WinMacro is just as old program, like Macro Dollar and still works great. Only 3 steps are required to record; specify the file into which recording will be made, click on the record button and click hotkey for Pause/Break to stop recording. Whenever you want to play back a recording, simply specify the path to the saved .log file and click the play button. Playback can be canceled using hot keys Ctrl keys+ Esc. The Options button allows you to set the number of repetitions and playback speed.
GhostMouse – ghostly simulation of mouse and button presses
GhostMouse is a very simple program for simulating user actions, which has only 2 buttons in the main window (Play & Record). It is more suitable for quick and short-term use, but, of course, also for automating repetitive tasks.
Unlike previous examples, GhostMouse is not portable and requires installation. In the Options menu, you can set the playback speed, as well as specify what the program should record or not record - keystrokes, mouse movements, and so on.
Do It Again – creating a list of macros
Do it again is not very different from the other 3 tools listed above, but it has one advantage - a list of macro records displayed in the program, which makes it convenient for users who need to run various tasks without manual loading script into the program.
Action(s) program – full automation of actions on the computer
Action(s) is a visual clone of Automator, which, however, offers slightly fewer options in the list of available interactions.
Automator- an ingenious built-in application for OS X that allows you to implement automation of actions on your computer without knowing any scripting language. The application may be useful for those who often work with digital photos. As a suitable example, you can indicate the need for an automatic action - copy all photos from a given folder to a new directory, reduce their resolution by 72 DPI and convert them from digital model CMYK to RGB. In addition, however, you can automate work with text, by email, PDF documents, set run scripts to specified time, and, for example, work with a SQL database.
They offer their own version both for Windows and, paradoxically, for OS X. You can install automation of actions on your computer with email, files and folders, images, browser and plain text.
The advantage is a ready-made set of macros that are most often used by users - mainly for working with photographs and audiovisual content. And if the desired action (macro) is not in the library, and you want to create it, then most likely the right solution can be found on the user forum dedicated to the Action(s) program. It is important to note that the program itself for automating your actions on personal computer or laptop is written in Java. There is support for simulating mouse movements and keyboard keystrokes.
Conclusion
In rare cases, I resort to the last lines of an article in the form of a conclusion, but here I decided to make exceptions.
Starting the article, I forgot to say that it is available not only in the Word, Excel editor, but also in graphic editor Photoshop. Yes, there is a function there that is, in some way, an analogue of a macro. So, you can record the actions that are used when processing a drawing, save them in the same PSD file, and then calmly reproduce for other graphic data. I believe there are similar opportunities in Corel Draw and in other similar programs.
Everyone, while playing, at least once thought: “I wish I could write a program that would play for me!” But usually this thought remains just a thought... Something constantly gets in the way: not knowing where to start, fear of the overwhelming task, a whisper from over your left shoulder “why is this? who needs it, for the program to play with the program?” etc.In this series of articles I am going to show that, firstly: “the devil is not as terrible as he is painted,” and secondly: later I am going to answer the question: “why is this necessary?”
Now let's start with something simple. From establishing the relationship between the game and the player program (bot). The well-known game Zuma is taken as a guinea pig.
Any interaction consists of two processes: sending data to “them” and receiving data from “them”. In Zuma, all controls are done with the mouse, and feedback the game displays using an image. Accordingly, first of all, you need to learn how to programmatically emulate the behavior of the mouse and obtain an image from the game.
The main goal of this article: to get a program that independently enters the game process over and over again, does something there, and when the game over-e starts all over again. Further, this framework will develop in the direction that the bot will last further and longer in the game until game over.
Solvable auxiliary subtasks: mouse emulation, mouse redirection to a virtual machine, image capture.
Retreat
When developing the code for this series of articles, the approach used is to get results as quickly as possible with a minimum of effort. This approach allows you to maintain motivation for high level, and does not allow one to give up when faced with the insurmountability of the task. Because of this:- firstly, many insignificant (from the point of view of the current result) points will be quickly skimmed over, leaving “crutches and supports” in the code. And only in the next iterations will these points be analyzed separately, and the “crutches” will be replaced with full-fledged code.
- secondly, the code style is more “hacker” than classic C#. The code will contain a lot of lambdas, anonymous data, tricks, authorial discretion and a complete lack of comments.
Mouse emulation
Windows supports 2 standard methods of mouse emulation using 4 different WinApi functions.First way: sending the program its own window messages (WM_MOUSEMOVE, WM_LBUTTONDOWN, etc.) using the SendMessage or PostMessage functions.
For DirectX games (as in our case), this method is not suitable, because such programs for polling the mouse use DirectInput, which polls the mouse directly, ignoring Windows messages.
Second way: Direct emulation of mouse behavior using the mouse_event or SendInput functions. This method is suitable for any programs, including full-screen DirectX games. The mouse_event function is simpler, but it is considered obsolete; SendInput is more modern, but more cumbersome. Let's focus on mouse_event.
WinApi functions from C# are called using PInvoke technology. PInvoke descriptions for most common WinApi functions can be found on the PInvoke.net website. The mouse_event function is no exception.
public static extern void mouse_event(uint dwFlags, int dx, int dy, uint dwData, UIntPtr dwExtraInfo);
Mouse coordinates
The mouse_event function has a special feature: the mouse coordinates are specified in mickey, not in pixels. Conversion of mickey to pixels (and vice versa) depends on the resolution of the main monitor used. (0,0) corresponds to left top corner monitor, and (65535, 65535) to the lower right, which gives formulas for converting mickey into pixels and vice versa: mickey_point = pixel_point * (65536, 65536) / screen_size and pixel_point = mickey_point * screen_size / (65536, 65536) .Basic Operations
Summarizing all of the above, we get the following operations for controlling the mouse.Moving the mouse cursor to point (x,y):
mouse_event(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE, x * 65536 / screen_width, y * 65536 / screen_height);
Left click:
mouse_event((MouseEventFlags.LEFTDOWN), 0, 0); System.Threading.Thread.Sleep(100); mouse_event((MouseEventFlags.LEFTUP), 0, 0);
Right click:
mouse_event((MouseEventFlags.RIGHTDOWN), 0, 0); System.Threading.Thread.Sleep(100); mouse_event((MouseEventFlags.RIGHTUP), 0, 0);
Problem: input exclusivity
When emulating a mouse through the mouse_event function, there is a serious inconvenience: mouse_event simulates a mouse for the entire OS at once, and not for separate application. It follows from this that while the bot is running and playing, other work on the computer is impossible: debugging the bot, actively viewing the bot’s state, reading the Internet, etc. But there is a way out: a virtual machine!Transferring a game to a virtual machine
Transferring the game to a virtual machine solves the following problems:- firstly, it simplifies interaction with games that do not support windowed mode and only work in full screen mode,
- secondly, mouse input is replaced only on the virtual machine, but on the main machine it continues to work in normal mode, allowing the computer user to go about his business.
The bot, unlike the game itself, is more convenient to run on the main machine. This allows you to restart the bot directly from Visual Studio, debug it there, have a place to display the internal state of the bot, etc.
Deployment virtual machine(in this case, Oracle VirtualBox was used), installing the guest OS and transferring the game is done as usual, with the exception of one point: the bot needs the ability to establish communication over the network between the host OS and the guest OS. This is done in many ways. One way is to forward a specific port from the guest OS to the host OS using VirtualBox. Another way is to configure the Bridged Adapter mode, then the virtual machine will look like a regular computer for the entire network, and the guest OS will receive its IP address via dhcp from the router. Access from the host OS to the guest OS will take place at this address. (the author, in this case, used the option with a bridged adapter)
Proxy
To control the mouse on the guest OS, we will write a proxy, which is a simple console TCP server. Its full code is small and is presented below the cut. To simplify the code and reduce dependencies, the proxy is written on a bare socket without using remoting, wcf, etc. Proxy code
using System; using System.Collections.Generic; using System.Linq; using System.Net.Sockets; using System.Runtime.InteropServices; using System.Text; namespace InputProxy ( class Program ( static void Main(string args) ( var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp); socket.Bind(new System.Net.IPEndPoint(System.Net.IPAddress. Any, 7001)); socket.Listen(10); for (; ;) ( var client = socket.Accept(); Console.WriteLine("connected.."); var thread = new System.Threading.Thread(( ) => ( try ( var clientReader = new System.IO.BinaryReader(new NetworkStream(client)); for (; ;) ( if (client.Poll(1, SelectMode.SelectRead) && client.Available == 0) ( Console.WriteLine("disconnected.."); break; ) if (client.Available > 0) ( var msgSize = clientReader.ReadInt32(); var message = clientReader.ReadBytes(msgSize); var messageReader = new System.IO. BinaryReader(new System.IO.MemoryStream(message)); var msgKind = messageReader.ReadInt32(); Console.WriteLine("message: kind:(0), len:(1)", msgKind, message.Length); switch (msgKind) ( case 0: ( var flags = messageReader.ReadUInt32(); var x = messageReader.ReadInt32(); var y = messageReader.ReadInt32(); var data = messageReader.ReadUInt32(); mouse_event(flags, x, y, data, UIntPtr.Zero); ) break; ) ) else System.Threading.Thread.Sleep(10); ) ) catch (Exception exc) ( Console.WriteLine(exc); ) )) ( IsBackground = true ); thread.Start(); ) ) public static extern void mouse_event(uint dwFlags, int dx, int dy, uint dwData, UIntPtr dwExtraInfo); ) )
For the proxy to work, you just need to copy it to a virtual machine and run it. The proxy waits for messages on port 7001 and displays a log of its operation on the console. To shut down the proxy, just close the console window.
Client
Connecting to a proxy is even simpler than the code for the proxy itself.var client = new System.Net.Sockets.TcpClient(vm_host, 7001); var clientStream = client.GetStream(); var clientWriter = new System.IO.BinaryWriter(clientStream); Action
Image interception
The easiest way to capture an image is directly from the screen. In .net there is a ready-made function for this, Graphics.CopyFromScreen. Let's look at this method in more detail.Firstly, you want to get a Bitmap as an output, not a Graphics - this can be solved using an auxiliary function:
public static Bitmap GetScreenImage(Rectangle rect) ( var bmp = new Bitmap(rect.Width, rect.Height, PixelFormat.Format32bppArgb); using ( Graphics graphics= Graphics.FromImage(bmp)) ( graphics.CopyFromScreen(rect.Left, rect.Top, 0, 0, rect.Size, CopyPixelOperation.SourceCopy); ) return bmp; )
Secondly, you need to know which part of the screen you need to capture. You can, of course, always capture the same part of the screen and place the game with your hands in this part of the screen, but this is not convenient or practical. Moreover, this process is being automated with minimal effort. WinApi and PInvoke will help us with this again, and more specifically two functions: FindWindow and GetWindowRect. FindWindow allows you to get the handle of the window using the window title, and GetWindowRect using the handle returns the position and size of the window on the screen.
There is a pinvoke description of both functions on the pinvoke.net website: FindWindow and GetWindowRect.
public static extern IntPtr FindWindow(string lpClassName, string lpWindowName); public static extern bool GetWindowRect(IntPtr hwnd, out RECT lpRect); public struct RECT ( public int Left; public int Top; public int Right; public int Bottom; )
And the code for capturing an image of a virtual machine window is as follows:
var vm_left = 8; var vm_right = 8; var vm_top = 50; var vm_bottom = 30; var vm_title = "Windows81 - Oracle VM VirtualBox"; var handle = FindWindow(null, vm_title); if (handle == IntPtr.Zero) throw new Exception("Окно не найдено"); RECT rect; GetWindowRect(handle, out rect); var gameScreenRect = new System.Drawing.Rectangle(rect.Left + vm_left, rect.Top + vm_top, rect.Right - rect.Left - vm_right - vm_left, rect.Bottom - rect.Top - vm_bottom - vm_top); var gameBmp = GetScreenImage(gameScreenRect); !}
Weakness
A significant drawback of this approach is that the captured window, firstly: must be entirely located on the screen, and secondly: must be located on top of all other windows. This inconvenience is leveled out with the help of two (or more) monitors :), then the virtual machine window is located on an auxiliary monitor, without disturbing anyone, remaining on top of the other windows. Also this problem can be completely solved using the previously discussed method: transferring the function (screen capture) inside the virtual machine. To do this, just add the appropriate function to InputProxy.Looping the gameplay
Finally, we proceed directly to solving the task set for today: looping the game process - all the necessary subtasks have been solved. The gameplay in Zuma revolves around three windows: main, mission and action. The main window contains the main menu, allowing you to select the type of game, the mission window prompts you to select a mission, and the action window is where the gameplay itself takes place.The bot determines the current window by the most in a simple way: by color value at several key points. Points are selected manually using the “close gaze” method.
var screenChecks = new ( new ( Name = "main", Points = new ( new CheckPoint(200, 190, 0xff554a22), new CheckPoint(65, 400, 0xfff44c41) ) ), new ( Name = "mission", Points = new ( new CheckPoint(200, 190, 0xffb5d0c7), new CheckPoint(65, 400, 0xffad7630) ) ), new ( Name = "action", Points = new ( new CheckPoint(950, 10, 0xff72554b), new CheckPoint(10, 10, 0xff462b1d), ) ), ); Func
The main bot loop:
var startButtonPoint = new Point(950, 430); var startMissionPoint = new Point(600, 750); for (; ;) ( try ( var bmp = GetScreenImage(gameScreenRect); var screenName = check(bmp); Console.Write(screenName + new string(" ", 20) + new string("\x8", 40)) ; switch (screenName) ( case "main": mouse_event(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE, startButtonPoint.X * 65536 / game_width, startButtonPoint.Y * 65536 / game_height); System.Threading.Thread.Sleep(400); mouse_event (MouseEventFlags.LEFTDOWN, 0, 0); System.Threading.Thread.Sleep(150); mouse_event(MouseEventFlags.LEFTUP, 0, 0); System.Threading.Thread.Sleep(50); System.Threading.Thread.Sleep (400); break; case "mission": mouse_event(MouseEventFlags.MOVE | MouseEventFlags.ABSOLUTE, startMissionPoint.X * 65536 / game_width, startMissionPoint.Y * 65536 / game_height); System.Threading.Thread.Sleep(10); mouse_event (MouseEventFlags.LEFTDOWN, 0, 0); System.Threading.Thread.Sleep(150); mouse_event(MouseEventFlags.LEFTUP, 0, 0); System.Threading.Thread.Sleep(50); break; case "action": mouse_event(MouseEventFlags.LEFTDOWN, 0, 0); System.Threading.Thread.Sleep(150); mouse_event(MouseEventFlags.LEFTUP, 0, 0); System.Threading.Thread.Sleep(50); break; case null: bmp.Save("unknown.bmp"); break; ) ) catch (Exception exc) ( Console.WriteLine(exc); ) )
During the game phase, the bot constantly clicks, releasing balls at one point. Using such simple (or rather stupid) tactics, the bot gains 1000-2000 points in the first mission, and sometimes even completely gains the Zuma bar.
Summary
The goal has been achieved: the bot framework has been written - the gameplay is looped.The following goals: connect OpenCV, recognize the position and color of the balls.
