How to scroll the vim completion popup window

After popup windows is added in vim 8.2, we can display documentation provided by ycm in a popup window. Different from display documentation in a preview window, it won't change the layout. But there is one disadvantage, I can't scroll the popup window by keyboard. It is not vim-style enough.

Enable the Completion Popup Window

Add the following line in the vimrc file.

set completeopt+=popup

Scroll the Popup Window

dedowsdi provide a solution about how to scroll a popup window here. There is another problem now. Not like coc, the popup window triggered by ycm may be far away from the cursor. If I follow this solution, I must let vim to scan over the whole screen to find the popup window. It is too slow.
After looking into VIM REFERENCE MANUAL and doing some tests, I thought I can use popup_findinfo to get the id of the completion popup window. But, I'm not sure it is the correct way. So I have the following code:

nnoremap <C-e> :call ScrollPopup(1)<CR>
nnoremap <C-y> :call ScrollPopup(0)<CR>

function ScrollPopup(down)
    let winid = popup_findinfo()
    if winid == 0
        return 0
    endif

    " The popup window has been hidden in the normal mode, we should make it show again.
    call popup_show(winid)
    let pp = popup_getpos(winid)
    call popup_setoptions( winid,
                \ {'firstline' : pp.firstline + ( a:down ? 1 : -1 ) } ) 

    return 1
endfunction

Then, I can read and scroll the completion popup window in normal mode. It isn't good enough still: 1. I must switch to normal mode which will close the completion popup menu. If the selection isn't the one I want, I must delete it and trigger completion again. 2. After scrolling to the bottom, the height of the popup window will reduce to one, if I keep pressing CTRL-e.

Scroll the Popup Window without Closing the Popup Menu

Firstly, I should change nnoremap to inoremap. So the key-binding will work in insert mode.

inoremap <C-e> :call ScrollPopup(1)<CR>
inoremap <C-y> :call ScrollPopup(0)<CR>

I haven't spend much time with vimscript even if I have used vim for years. Instead of calling ScrollPopup, it just inserts ":call ScrollPopup(1)\n". Finally, I find out how map works.

map {lhs} {rhs}

It just maps the left hand side keystrokes to the right hand side keystrokes. If I want to run something, I should add <expr>.

map <expr> {lhs} {rhs}

The {rhs} is evaluated to obtain the right hand side keystrokes.

inoremap <expr> <C-e> ScrollPopup(3) ? '' : '<C-e>'
inoremap <expr> <C-y> ScrollPopup(-3) ? '' : '<C-y>'

If scroll happened, it maps CTRL-e/CTRL-y to non-operation. Otherwise, it doesn't change the keystrokes.

Stop after Reaching the bottom

ScrollPopup should stop changing firstline after the bottom of the buffer has showed. But, how does it know that? With popup_getpos, we have following properties:

col         screen column of the popup, one-based
line        screen line of the popup, one-based
width       width of the whole popup in screen cells
height      height of the whole popup in screen cells
core_col    screen column of the text box
core_line   screen line of the text box
core_width  width of the text box in screen cells 
core_height height of the text box in screen cells 
firstline   line of the buffer at top (1 unless scrolled) (not the value of the "firstline" property)
lastline    line of the buffer at the bottom (updated when the popup is redrawn)
scrollbar   non-zero if a scrollbar is displayed
visible     one if the popup is displayed, zero if hidden

At a glance, a formula has come up: lastline - firstline < height. But it doesn't work. lastline is not the lastline of the text. It is the lastline showed in the popup window. After some search, I found I can have the lastline by win_execute(winid, "echo line('$')"). Executing a command in the popup window to get the lastline of the buffer.
It still doesn't work as I thought. Sometimes, it won't scroll down. Occasionally, I show the line number in the popup window. firstline and lastline is the line of the text. height is the line of the window. With wrap on, one line of text will be many lines in the window. Finally, I have:

function! ScrollPopup(down)
    let winid = popup_findinfo()
    if winid == 0
        return 0
    endif

    " if the popup window is hidden, bypass the keystrokes
    let pp = popup_getpos(winid)
    if pp.visible != 1
        return 0
    endif

    let firstline = pp.firstline + a:down
    let buf_lastline = str2nr(trim(win_execute(winid, "echo line('$')")))
    if firstline < 1
        let firstline = 1
    elseif pp.lastline + a:down > buf_lastline
        let firstline = firstline - a:down + buf_lastline - pp.lastline
    endif

    " The appear of scrollbar will change the layout of the content which will cause inconsistent height.
    call popup_setoptions( winid,
                \ {'scrollbar': 0, 'firstline' : firstline } )

    return 1
endfunction

inoremap <expr> <C-e> ScrollPopup(3) ? '' : '<C-e>'
inoremap <expr> <C-y> ScrollPopup(-3) ? '' : '<C-y>'