Android Control Skill

Control Android phones via ADB. Prerequisite: phone has USB debugging enabled and is connected to the computer.

Check Connection

adb devices

If it shows unauthorized, you need to allow USB debugging on the phone.

Common Commands

Screenshot

# Method 1: Screenshot to phone then pull
adb shell screencap -p /sdcard/screenshot.png
adb pull /sdcard/screenshot.png ./
adb shell rm /sdcard/screenshot.png

# Method 2: Direct output to local (recommended)
adb exec-out screencap -p > screenshot.png

Tap

adb shell input tap <x> <y>

# Example: tap center of screen (1080x2400 resolution)
adb shell input tap 540 1200

Swipe

adb shell input swipe <x1> <y1> <x2> <y2> <duration_ms>

# Swipe up (finger moves from bottom to top)
adb shell input swipe 500 1500 500 500 300

# Swipe down
adb shell input swipe 500 500 500 1500 300

Input Text

adb shell input text "hello"

Note: Only supports English characters. For Chinese, use URL parameters or clipboard approach.

Key Events

# Common key events
adb shell input keyevent KEYCODE_HOME      # Home key
adb shell input keyevent KEYCODE_BACK      # Back key
adb shell input keyevent KEYCODE_APP_SWITCH # Recent tasks
adb shell input keyevent KEYCODE_POWER     # Power key
adb shell input keyevent KEYCODE_WAKEUP    # Wake screen
adb shell input keyevent KEYCODE_VOLUME_UP # Volume +
adb shell input keyevent KEYCODE_VOLUME_DOWN # Volume -

Get Screen Resolution

adb shell wm size

Open App

adb shell am start -n <package>/<activity>

# Common apps
adb shell am start -n com.tencent.mm/.ui.LauncherUI           # WeChat
adb shell am start -n com.android.camera/.Camera              # Camera
adb shell am start -n com.android.mms/.ui.ComposeMessageRouterActivity # SMS

Get Current Activity

adb shell dumpsys window | grep -i mCurrentFocus

Get UI Element Info

# Export UI hierarchy
adb shell uiautomator dump /sdcard/ui.xml
adb shell cat /sdcard/ui.xml

# Find specific element
adb shell cat /sdcard/ui.xml | grep "send_button"

AI Operation Flow

1. Screenshot first, analyze the interface with image tools
2. Determine the coordinates to tap/operate
3. Execute the operation
4. Screenshot again to verify results
5. Loop until the task is complete

MIUI Special Handling

Camera Photo Capture

Xiaomi phones using android.media.action.IMAGE_CAPTURE Intent will open OneShotCamera mode, requiring manual confirmation to save. It's recommended to use normal camera mode:

# Open normal camera (auto-save)
adb shell am start -n com.android.camera/.Camera

# Wait for camera to launch
sleep 2

# Tap the shutter button (bottom center of screen, 1080x2400 resolution)
adb shell input tap 540 2100

# Wait for photo capture to complete (MIUI auto-saves to DCIM/Camera/)
sleep 2

Notes:

• Normal camera mode auto-saves photos without manual confirmation
• Photos are saved in /storage/emulated/0/DCIM/Camera/

Get Latest Photo

# Query latest photo
adb shell "content query --uri content://media/external/images/media 2>/dev/null | grep -E '(DCIM|Camera)' | head -1"

# Pull photo to local
adb pull /storage/emulated/0/DCIM/Camera/IMG_xxx.jpg ./photo.jpg

SMS Sending

MIUI SMS app has special handling:

# Open SMS compose interface via Intent (supports URL-encoded Chinese content)
PHONE="<phone number>"
BODY=$(echo 'SMS content' | jq -sRr @uri)
adb shell "am start -a android.intent.action.SENDTO -d 'sms:${PHONE}?body=${BODY}'"

# Wait for interface to load
sleep 2

# Send button position (1080x2400 resolution)
# First get button coordinates
adb shell uiautomator dump /sdcard/ui.xml
adb shell cat /sdcard/ui.xml | grep 'send_button' | grep -oE 'bounds="\[[0-9,]+\]\[[0-9,]+\]"'

# Tap send button (adjust based on actual coordinates)
adb shell input tap 992 2282

Important notes:

• Content is passed via URL parameters, supports Chinese
• The send button only becomes a "Send SMS" icon when the input field has content
• Permission requests (microphone, etc.) may pop up and need to be handled

Handling Permission Dialogs

# Use uiautomator to get button position
adb shell uiautomator dump /sdcard/perm.xml
adb shell cat /sdcard/perm.xml | grep -E "(允许|拒绝|Allow|Deny)" | grep -oE 'text="[^"]*"|bounds="[^"]*"'

# Tap the corresponding button
adb shell input tap <x> <y>

Screenshot Restrictions

MIUI may block screenshots on certain screens (returns a 15KB empty file). Solution:

# Use exec-out method (recommended)
adb exec-out screencap -p > screenshot.png

# Check if screenshot is valid (should be >100KB)
ls -la screenshot.png

Notification Shade Cannot Be Swiped Closed

MIUI notification shade may not close via swipe, use:

adb shell input keyevent KEYCODE_HOME

Tap Requires Additional Permissions

If tap gives error INJECT_EVENTS permission: Settings → Additional Settings → Developer Options → USB Debugging (Security Settings) → Enable

Utility Functions

Send SMS

send_sms() {
    local phone="$1"
    local message="$2"
    local body=$(echo "$message" | jq -sRr @uri)
    
    adb shell "am start -a android.intent.action.SENDTO -d 'sms:${phone}?body=${body}'"
    sleep 2
    
    # Get send button coordinates and send
    adb shell uiautomator dump /sdcard/ui.xml
    local bounds=$(adb shell cat /sdcard/ui.xml | grep 'content-desc="发送短信"' | grep -oE 'bounds="\[[0-9,]+\]\[[0-9,]+\]"' | head -1)
    
    if [ -n "$bounds" ]; then
        # Parse coordinates and tap center point
        local x1=$(echo "$bounds" | grep -oE '\[([0-9]+),' | head -1 | tr -d '[,' )
        local y1=$(echo "$bounds" | grep -oE ',([0-9]+)\]' | head -1 | tr -d ',]')
        local x2=$(echo "$bounds" | grep -oE '\[([0-9]+),' | tail -1 | tr -d '[,' )
        local y2=$(echo "$bounds" | grep -oE ',([0-9]+)\]' | tail -1 | tr -d ',]')
        local cx=$(( (x1 + x2) / 2 ))
        local cy=$(( (y1 + y2) / 2 ))
        adb shell input tap $cx $cy
        echo "SMS sent to $phone"
    else
        echo "Send button not found"
    fi
}

# Usage example
# send_sms "13800138000" "This is a test SMS"

Take Photo and Retrieve

take_photo() {
    local output="${1:-photo.jpg}"
    
    adb shell am start -n com.android.camera/.Camera
    sleep 2
    adb shell input tap 540 2100  # Shutter button
    sleep 2
    
    # Get the latest photo
    local latest=$(adb shell "content query --uri content://media/external/images/media 2>/dev/null | grep 'DCIM/Camera' | head -1" | grep -oE '/storage[^,]*')
    
    if [ -n "$latest" ]; then
        adb pull "$latest" "$output"
        echo "Photo saved to $output"
    else
        echo "Photo not found"
    fi
}

# Usage example
# take_photo "my_photo.jpg"

ADBKeyboard Chinese Input

adb shell input text does not support Chinese. ADBKeyboard must be installed:

# Install ADBKeyboard
curl -L -o /tmp/ADBKeyboard.apk "https://github.com/senzhk/ADBKeyBoard/raw/master/ADBKeyboard.apk"
adb install /tmp/ADBKeyboard.apk

# Enable and set as default input method
adb shell ime enable com.android.adbkeyboard/.AdbIME
adb shell settings put secure default_input_method com.android.adbkeyboard/.AdbIME

# Input Chinese (input field must have focus)
adb shell am broadcast -a ADB_INPUT_TEXT --es msg 'This is Chinese content'

# Restore original input method (e.g., iFlytek)
adb shell settings put secure default_input_method com.iflytek.inputmethod.miui/.FlyIME

⚠️ Important Rules

Failure Retry Limit

Same operation fails more than 3 times → stop immediately and report; do not retry.

This includes:

• Same coordinate tapped 3 times with no response
• Same command executed 3 times and failed
• Same UI element not found after 3 searches

In such cases, you should:

1. Stop the current operation
2. Report the failure reason to the user
3. Provide possible solutions or ask for next steps

Chinese Input (ADBKeyboard)

ADB's native input text does not support Chinese. Solution: install ADBKeyboard.

Installation & Configuration

# Download and install
curl -L -o /tmp/ADBKeyboard.apk "https://github.com/senzhk/ADBKeyBoard/raw/master/ADBKeyboard.apk"
adb install /tmp/ADBKeyboard.apk

# Enable and set as default input method
adb shell ime enable com.android.adbkeyboard/.AdbIME
adb shell settings put secure default_input_method com.android.adbkeyboard/.AdbIME

Usage

# Input Chinese (must tap input field to gain focus first)
adb shell am broadcast -a ADB_INPUT_TEXT --es msg 'This is Chinese content'

# Restore original input method (iFlytek)
adb shell settings put secure default_input_method com.iflytek.inputmethod.miui/.FlyIME

Note: You must ensure the input field has focus (has been tapped and activated) before inputting.

Notes

• Coordinates are based on the phone's actual resolution (common for Xiaomi: 1080x2400)
• input text does not support Chinese — use ADBKeyboard or URL parameter approach
• Some apps may have anti-automation detection
• MIUI camera normal mode auto-saves photos; OneShotCamera mode requires manual confirmation
• MIUI blocks screenshots on certain screens, returning a 15KB empty file
• Use uiautomator dump to get accurate UI element coordinates
• Same operation fails more than 3 times → stop and report